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FOREWORD 


The  School  of  Applied  Aerospace  Sciences  at  Chanute  AFB, 

Illinois,  was  the  first  military  training  center  to  undertake  I 

an  extensive  investigation  of  the  use  of  the  PLATO  IV  system 
in  technical  training.  This  research  was  sponsored  by  the 
Advanced  Research  Projects  Agency  (ARPA)  of  the  Department  of 
Defense  and  the  Air  Training  Command  (ATC)  of  the  United 
States  Air  Force.  ARPA  provided  hardware  and  hardware  sup- 
port while  ATC  staffed  the  project. 

It  was  originally  intended  that  the  project  have  25  \ 

PLATO  IV  terminals.  However,  by  the  project's  end  30  term-  i 

inals  were,  and  still  are,  in  place  at  Chanute.  These  extra  ; 

terminals  were  necessitated  by  the  practical  demands  of  using 
the  PLATO  system  as  an  operational  medium  of  instruction. 

Each  of  these  terminals  is  equipped  with  a graphic  display 

panel,  touch  panel,  and  microfiche  slide  selector.  j 

Historical  Summary 

ARPA  funding  began  in  July,  1972,  and  ceased  in  June,  j 

1976.  During  this  time,  the  service  test  went  through  four 
distinct  phases.  Each  phase  was  characterized  by  distinct  / 

modes  of  management  and  objectives. 

Phase  one.  The  initial  phase  of  the  service  test  was 
characterized  by  uncertainty  in  leadership,  objectives  and 
orientation.  The  leadership  and  objectives  both  underwent 
several  changes  during  this  phase.  Likewise,  the  orientation 
of  the  project  was  initially  towards  producing  courseware 
for  a large  scale  test  of  computer-based  technical  training. 

After  some  time  this  was  given  up  and  interest  shifted  from 
a primarily  operational  emphasis  to  one  of  research.  This 
vacillation  of  the  project  was  probably  a result  of  lack  of 
active  interest  on  the  part  of  the  responsible  Air  Force 
agencies . 


Phase  two.  When  a more  active  interest  in  the  service 
test  was  taken,  the  project  was  put  in  the  hands  of  an  exper- 
ienced curriculum  designer  and  was  given  specific  objectives 
During  this  phase,  CBE  materials  were  to  be  produced  for  use 
in  operational  training.  Any  research  objective  became  in- 
cidental to  that  of  producing  courseware  for  Air  Force  train- 
ees. By  the  modification  of  some  materials  from  the  first 
phase  and  creating  the  rest,  this  objective  was  met.  The 
use  of  the  resulting  CBE  materials  by  Air  Force  trainees  has 
resulted  in  a rich  source  of  data  concerning  the  use  of  CBE 
in  a military  training  environment.  This  phase  came  to  an 
end  in  June,  1975. 

Phase  three.  Following  the  validation  and  implementa- 
tion of  these  materials,  the  service  test  undertook  some 
small  researches  and  courseware  development  efforts.  An 
evaluation  of  this  latter  part  is  found  in  Chapter  IX  of 
Dallman  ^ al . ( 1 977 ) . 

Phase  four.  When  ARPA  funding  stopped  in  June,  1976, 

ATC  funded  a brief  follow-on  study  of  three  months.  During 
this  period  information  was  to  be  gathered  concerning  spe- 
cific benefits  of  PLATO  use.  These  studies  focused  on  the 
cost  and  times  savings  due  to  PLATO  use,  the  efficiency  of 
PLATO-based  testing,  and  the  volume  of  trainess  that  could  be 
trained  with  the  PLATO  courseware  developed  during  the  first 
two  phases.  This  study  was  completed  in  October,  1976.  The 
three  studies  of  this  phase  are  reported  in  Main  (1977). 

The  information  gathered  in  the  fourth  phase  provided  a 
basis  for  the  Chanute  Center  Commander  to  request  funding 
from  ATC  for  the  use  of  PLATO  as  an  operational  training  med- 
ium at  Chanute  AFB.  This  funding  was  provided  so  that  the 
30  PLATO  terminals  presently  located  at  Chanute  are  currently 
delivering  computer-based  technical  training  to  Air  Force 
trainees.  This  training  largely  consists  of  PLATO  lessons 


developed  during  the  first  two  phases  of  the  service  test. 

They  are  now  being  used  at  a approximate  rate  of  7000  stud- 
ent contact  hours  per  year  with  an  estimated  yearly  flow  of 
350  trainees. 

This  volume  is  concerned  almost  exclusively  with  the 
first  two  phases  of  the  service  test.  This  period  of  the 
service  test  accounted  for  almost  all  of  the  CBE  courseware 
and  most  of  its  data  on  the  use  of  CBE  in  a military  training 
environment.  In  addition  to  a wealth  of  data,  these  phases 
present  some  very  valuable  lessons  in  the  management  and 
administration  of  such  a service  test.  Because  of  the 
importance  of  these  lessons  and  data  and  because  the  later 
phases  of  the  service  test  did  not  add  substantially  to  what 
was  already  known,  it  seemed  appropriate  to  focus  attention 
on  the  first  two  phases. 

The  Critique  and  Summary  of  the  Service  Test 

Part  One.  Part  One  gives  a fairly  detailed  summary  of 
the  history  of  the  service  test's  first  two  phase.  During 
the  first  phase,  the  service  test  floundered.  The  sources  of 
the  service  test  difficulties  both  at  the  site  of  the  test 

and  from  ATC  and  AFHRL  would  be  evident  to  any  casual  obser- 

ver of  the  service  test.  Neverthless,  in  such  matters  it  is 
always  good  to  have  one's  intuitions  about  a situation  con- 
firmed. The  second  phase  of  the  service  test  serves  this 
\ purpose.  At  the  beginning  of  that  phase,  most  of  the  appar- 

' ent  shortcomings  of  the  first  phase  were  eliminated.  The 

' outcome  of  the  second  phase  has  already  been  mentioned. 

Part  Two.  Part  Two  is  a commentary  on  the  Evaluation 
' of  PLATO  IV  in  Vehicle  Maintenance  Training  (Dallman  et  al . , 

1977).  Part  Two  is  intended  to  be  read  in  this  evaluation 
report.  Its  Foreword  (p.  62)  describes  its  purpose  and  scope. 
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ESTABLISHMENT  OF  THE  AIR  FORCE 

Research  Agreement 

The  first  research  agreement  that  governed  the  United 
States  Air  Force's  service  test  of  the  PLATO  IV  system  was 
signed  by  the  involved  agencies  in  July,  1972.  The  pro- 
jected three  year  service  test  was,  in  the  words  of  the 
agreement,  to  determine  the  "cost  effectiveness,  acceptance, 
and  incorporabil ity  (into  on-going  training),  and  reliabil- 
ity" of  computer-based  education  (CBE)  in  a military  train- 
ing environment.  The  research  agreement  also  outlined  the 
responsibilities  of  the  agencies  that  would  be  involved  in 
the  service  test.  These  agencies  were  the  Air  Training 
Command  (ATC),  the  Air  Force  Systems  Command  Human  Resources 
Laboratory  (AFHRL),  the  Advanced  Research  Projects  Agency 
(ARPA)  of  the  Department  of  Defense,  and  the  Computer-based 
Education  Research  Laboratory  (CERL)  of  the  University  of 
Illinois.  The  following  summary  of  the  responsibilities  of 
each  of  these  organizations  is  abstracted  from  this  research 
agreeement . 

The  ATC  School  of  Applied  Aerospace  Sciences  at  Chanute 
Air  Force  Base,  Illinois,  was  designated  as  the  site  of  the 
service  test.  At  that  site,  ATC  was  to  provide  space  to 
house  25  PLATO  IV  terminals,  students  for  both  the  instruc- 
tional materials  in  the  conventional  media  and  the  parallel 
computer-based  materials  to  be  written,  and  a staff  of  eight 
full-time  members  to  write  and  implement  the  computer-based 
materials.  The  task  of  documenting  these  materials  and 
their  production  was  also  given  to  ATC.  Except  for  the 
PLATO  IV  terminals  and  their  communications  costs,  ATC  sup- 
plied almost  all  of  the  physical  and  personnel  resources 
for  the  test. 


Through  a contract  with  the  University  of  Illinois, 

ARPA  furnished  those  resources  which  were  not  provided  by 
ATC.  ARPA's  support  consisted  of  the  installation  and  main- 
tenance of  25  PLATO  IV  terminals  along  with  the  funding  of 
communications  to  and  time  on  the  PLATO  IV  system’s  computer 
located  at  CERL.  The  terminals  were  to  be  equipped  with 
slide  selectors,  touch  panels,  and  audio  devices  so  that  a 
full  capability  for  computer-based  education  would  be 
available  for  the  service  test.  ARPA  also  provided  funds 
for  CERL  to  train  ATC  and  AFHRL  personnel  in  the  TUTOR 
language,  curriculum  development,  operating  dynamics,  and 
instructional  capabilities  of  the  PLATO  IV  system.  The 
Military  Training  Centers  (MTC)  group  at  CERL  was  to  carry 
out  the  initial  training  for  the  Air  Force  service  test  and 
give  advisory  assistance  in  curriculum  development  and  use 
of  the  PLATO  IV  system  for  the  duration  of  the  project. 

The  remaining  major  participant  in  the  service  test  was 
AFHRL.  This  laboratory  is  responsible  for  all  Air  Force 
research  in  the  realm  of  behavioral  sciences  and  personnel 
resources.  Thus  the  task  of  providing  overall  guidance  and 
of  designing  and  carrying  out  an  evaluation  of  the  service 
test  naturally  fell  to  AFHRL.  Specifically,  according  to 
the  research  agreement,  AFHRL  was  responsible  for: 

1)  providing  direction  and  monitoring  of  the 
service  test, 

2)  deriving,  implementing,  and  monitoring  mea- 
sures of  student  and  Instructor  attitudes 
as  well  as  performance, 

3)  consulting  on  instructional  systems  develop- 
ment, Instructional  materials,  and  media 
devices , 

4)  consulting  on  the  Incorporation  of  PLATO  IV 
into  an  on-going  training  environment  and  on 
the  use  of  PLATO  and  the  TUTOR  programming 
language , 
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5)  serving  as  the  primary  point  of  contact  to 
AREA  on  all  matters  pertaining  to  the  Univer- 
sity of  Illinois  contract, 

6)  providing  quarterly  letter  reports  of  pro- 
gress, an  annual  summary  report  of  progress 
and  findings,  and  a final  report  at  the  con- 
clusion of  the  experiment. 

Located  at  Lowry  AFB,  Colorado,  AFHRL  is  about  900  miles 
from  Rantoul,  Illinois,  the  location  of  Chanute  AFB.  Thus, 
AFHRL  was  tasked  with  direction  of  a project  in  which  it  had 
almost  no  investment  in  material  resources  from  a distance 
that  made  vigilant  monitoring  and  accurate  communications 
difficult  at  best. 

The  service  test  to  which  these  organizations  were  to 
contribute  was  orginally  intended  to  compare  a convention- 
ally taught  course  with  computer-based  instructional  mate- 
rials. The  General  Purpose  Vehicle  Maintenance  Course 
(GPVC)  (3ABR47330)  was  selected  for  the  purposes  of  this 
comparison. 1 This  course  teaches  a variety  of  skills 
including  inspecting,  servicing,  testing,  adjusting, 
trouble-shooting,  and  repairing  general  purpose  automotive 
vehicl  ..  such  as  light  trucks  and  automobiles.  The  variety 
of  topi  ^ jovered  in  the  course  indicated  that  it  would 
provide  a good  opportunity  for  the  exploration  of  the 

^The  verbal  designation  of  this  course  from  service 
test  documents  is  unclear.  Some  refer  to  the  course  as  the 
General  Purpose  Motor  Vehicle  Maintenance  Course;  others,  as 
the  General  Purpose  Vehicle  Repairman  Course;  and  still 
others,  as  the  General  Purpose  Vehicle  Maintenance  Course. 
Despite  the  disagreement  on  names,  all  documents  agree  that 
the  course  is  officially  designated  by  3ABR47330.  For  sim- 
plicity of  reference  here  the  course  will  be  designated 
GPVC  for  "General  Purpose  Vehicle  Course.” 
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potentials  and  limitations  of  computer-based  education.  The 
course,  as  it  was  conventionally  taught,  primarily  employed 
programmed  texts  and  took  12  weeks  for  students  to  complete, 
the  projected  computer-based  course  was  intended  to  dupli- 
cate the  objectives  of  the  existing  course  point  for  point 
while  teaching  them  in  a self-paced  mode.  By  self-pacing 
the  course,  it  was  expected  that  substantial  time  savings 
would  be  realized. 

The  eight  staff  members  who  were  to  develop  the 
computer-based  materials  were  to  be  under  the  authority  of 
the  GPVC  supervisor.  By  having  charge  of  both  the  experi- 
mental and  conventional  courses,  it  was  thought  that  he 
could  insure  that  the  computer-based  materials  taught  the 
same  objectives  as  the  conventional  materials.  In  addition 
to  the  guidance  of  the  course  supervisor,  the  Training 
Research  Applications  Branch  (TRAB),  the  School's  evaluative 
branch,  held  the  responsibility  jointly  with  AFHRL  of  devis- 
ing an  implementation  and  evaluation  plan  for  the  service 
test.  Thus,  the  PLATO  staff  was  under  the  influence  of 
three  distinct  sources--the  GPVC  chief,  the  TRAB,  and  AFHRL. 
Fragmentation  of  the  authority  over  the  PLATO  service  test 
later  contributed  to  several  of  the  difficulties  that 
plagued  the  project. 

The  research  agreement  was  approved  in  June,  1972.  The 
service  test  itself  began  on  15  July  1972  and  was  to  con- 
tinue until  15  July  1975.  The  final  report  for  the  test  was 
due  15  January  1976. 

Attempts  to  Establish  a Service  Test  Implementation  Plan 

For  the  first  21  months  of  the  service  test,  there  was 
no  finally  approved  implementation  plan.  'he  absence  of 
such  a plan  caused  several  management  problems  for  the 
administrators  of  the  service  test  which  will  be  discussed 


i 
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in  the  section  covering  project  management  before  May,  197^. 
In  discussing  the  establishment  of  the  service  test,  a major 
concern  will  be  to  give  a brief  description  and  chronology 
of  the  several  tentative  implementation  plans  the  project 
labored  under. 

First  Plan.  The  first  implementation  plan  was  a simple 
elaboration  of  the  research  agreement.  It  was  submitted  by 
TRAB  to  AFHRL  for  review  and  comments  on  November  9,  1972. 

It  called  for  the  development  of  a PLATO  based  course  paral- 
lel in  subject  matter  to  the  already  existing  GPVC.  The  two 
courses  would  then  be  compared  with  respect  to  grades,  reme- 
dial instruction  times,  course  repetition  and  failure  rates 
along  with  a variety  of  ad  hoc  measures  of  instructional  and 
cost  effectiveness. 

The  major  administrative  responsibility,  both  in  daily 
matters  and  for  execution  of  an  implementation  plan,  resided 
with  the  GPVC  supervisor.  While  the  eight  staff  members  who 
were  selected  to  develop  the  PLATO  materials  were  directly 
under  his  authority,  TRAB  also  exercised  some  influence  on 
the  PLATO  staff  in  an  advisory  capacity.  Because  the  GPVC 
supervisor's  interest  in  the  PLATO  service  test  was  not 
great,  TRAB  took  a more  active  part  in  the  test  than  had 
been  anticipated. 

An  active  role  for  TRAB  in  the  PLATO  service  test  arose 


originally  from  the  fact  that  TRAB's  branch  chief  was  the 
originator  of  the  Request  for  Training  Research  (RTR)  which 
led  to  the  Air  Force's  PLATO  service  test.  His  was  the  plan 
that  called  for  the  computer-based  versus  conventionally 
based  instruction  comparison  (Whalen,  1972).  Being  in  some 
sense  responsible  for  the  existence  of  the  service  test, 

TRAB  naturally  took  an  interest.  This  interest  combined 
with  its  joint  responsibility  with  AFHRL  for  an  implementa- 
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supervisory  role  that  TRAB  played  in  the  CDE  project.  More- 
over, the  fact  that  there  was  a TRAB  office  on  the  base  at 
Chanute  allowed  it  to  monitor  progress  of  the  project  more 
closely  than  the  more  experienced  supervisory  personnel  at 
AFHRL. 

For  four  months  from  the  submission  of  the  first  imple- 
mentation plan--from  November,  1972  to  January,  1973 — TRAB 
personnel  and  AFHRL  project  monitors  discussed  the  plan. 

They  were  unable  to  settle  on  a final  version  of  the  plan 
during  this  time.  At  the  end  of  this  period,  the  concept  of 
implementing  the  service  test  as  suggested  by  the  RTR  was 
questioned  and,  finally,  rejected. 

The  reasons  for  giving  up  the  first  implementation  plan 
involve  problems  with  the  start-up  of  the  project,  naivet* 
in  drawing  up  the  original  RTR,  and  changes  in  the  available 
student  populations  at  Chanute.  The  first  problem  relates 
primarily  to  management  of  the  project  and,  hence,  can  be 
treated  most  appropriately  in  the  section  discussing  the 
service  test's  management  before  May,  1974.  It  is  enough  to 
note  here  that  the  problems  in  getting  the  test  underway 
were  alone  sufficient  to  make  the  first  plan  appear 
extremely  difficult  to  complete  at  best. 

The  outline  of  the  service  test  set  forth  in  the  RTR 
was  not  thoroughly  researched.  While  it  was  attractive  at 
first  to  compare  directly  conventional  and  computer-based 
instruction,  it  appeared  later  that  such  comparison  was  not 
desirable  because  of  the  nature  of  the  conventional  GPVC. 
This  course  was  taught  primarily  through  the  use  of  pro- 
grammed texts.  Hence,  if  the  computer-based  materials  were 
to  be  developed  so  that  they  were  comparable  for  evaluation 
purposes  to  the  conventional  materials,  they  would  not  give 
any  demonstration  of  the  capabilities  of  CBE  beyond  the 
evident  capability  for  simulating  programmed  texts. 
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Moreover,  since  the  conventional  GPVC  was  already  effec- 
tively taught  with  programmed  texts,  a medium  much  less 
expensive  than  CBE,  all  hope  of  studying  the  cost  effective- 
ness of  CBE  would  have  to  be  given  up.  (Recall  that  deter- 
mination of  cost  effectiveness  was  one  of  the  service  test's 
original  objectives.)  Alternatively,  if  the  PLATO-based 
materials  were  written  with  a view  towards  fully  using  the 
PLATO  system's  capabilities,  the  resulting  materials  would 
differ  so  much  from  the  conventional  that  a comparison  of 
the  instructional  effectiveness  of  the  two  media  would  be 
impossible.  This  was  the  major  theoretical  difficulty  with 
the  original  plan  for  the  service  test. 

As  the  Chanute  service  test  got  underway,  other  practi- 
cal obstacles  to  the  successful  implementation  of  the  origi- 
nal comparative  evaluation  concept  appeared.  The  greatest 
of  these  was  a sudden  decrease  in  the  number  of  students 
taking  the  GPVC.  In  December,  1972,  it  appeared  that  there 
would  not  be  enough  students  taking  the  course  to  provide 
data  for  significant  comparison  of  the  conventional  and 
computer-based  courses. 

Finally,  the  service  test  began  while  the  PLATO  IV 
system  was  being  built  up  to  its  present  configuration. 

Thus,  problems  that  the  managers  of  the  service  test  had  in 
organizing  and  managing  their  project  were  compounded  with 
frequent  breaks  in  computer  service  of  duration  from  a few 
minutes  to  several  days  and  by  delays  in  delivery  of 
hardware . 

These  problems,  and  others  which  will  be  detailed  later 
in  this  section  and  in  the  following  section  on  early  pro- 
ject management,  convinced  the  administrators  at  Chanute 
that  the  original  conception  of  the  service  test  was  infea- 
sible. Simultaneously,  the  monitors  at  AFHRL  also  began  to 
question  the  original  plan.  According  to  the  One  Year 
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Interim  Report  written  by  members  of  TRAB,  these  monitors 
felt  that  the  two  years  that  were  to  be  spent  developing 
materials  for  the  GPVC  could  be  better  spent  in  answering 
specific  questions  concerning  computer-assisted  instruction 
(Green,  1973,  p.2). 

Second  plan.  On  January  5,  1973,  TRAB  sent  to  AFHRL  a 
memorandum  outlining  a new  implementation  plan  for  the  ser- 
vice test.  The  new  plan  would  replace  the  time  that  was  to 
be  spent  in  developing  materials  for  the  GPVC  with  research 
into  specific  questions  concerning  the  applications  of  CBF 
to  technical  training.  According  to  the  memorandum,  this 
research  would  emphasize  "direct,  simplified,  short-term 
research  studies"  (Main,  1973a,  p.1).  As  an  indication  of 
the  scope  of  the  research  being  contemplated,  the  following 
research  questions  are  excerpted  from  the  memorandum  (Ibid., 
pp.  2-3): 

Can  PLATO  serve  as  a part-task  training  device 
in  place  of  equipment? 

Is  PLATO'S  ability  to  adapt  to  a variety  of 
instructional  strategies  an  asset  in  Air  Force 
technical  training? 

Does  the  ability  to  employ  drill  and  practice, 
tutorial,  page  turning,  or  copy  frames  all  within 
the  same  lesson  enhance  learning  efficiency? 

Can  PLATO  be  programmed  to  adapt  to  the  aptitude 
of  the  student  going  through  the  lesson? 

Is  the  microfiche  display  an  effective  add-on 
to  the  capabilities  of  the  CBE  system?  For  what 
types  of  learning  or  performance  behaviors  are 
microfiche  slides  useful  or  necessary? 

To  study  these  and  similar  questions,  a plan  for  the  utili- 
zation of  the  project's  resources  was  offered.  The  plan  was 
an  attempt  to  simultaneously  answer  some  basic  research 
questions  about  CBE  while  obtaining  information  about 
management  and  cost  aspects  of  using  CBE  as  an  operational 
instructional  medium.  It  thus  represented  an  attempt  to 
plan  a service  test  which  would  satisfy  the  interests  of 
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both  AFHRL  and  ATC. 

This  memorandum  led  to  a meeting  between  representa- 
tives of  the  TRAB  and  AFHRL  to  set  up  guidelines  for  the 
implementation  of  new  ideas  for  the  service  test.  At  this 
meeting  the  following  points  were  agreed  upon  (Green,  1973> 
p • 3) : 

1)  The  TRAB  would  be  responsible  for  the  day- 
to-day  operation,  and  HRL  (AFHRL]  would 
act  as  a second- level  supervisory  body  and 
would  publish  the  service  test  plan. 

2)  There  would  be  investigation  of  the  use 
of  PLATO  both  for  computer-aided  instruc- 
tion and  computer-managed  instruction  (CMI). 

3)  Approximately  80  hours  of  computer  instruc- 
tion would  be  developed  for  CERL  community 
college  application. 

No  attempt  would  be  made  to  incorporate 
PLATO  into  the  Motor  Vehicle  Maintenance 
Course  through  a formal  ISD  effort. 

5)  Due  to  the  inability  to  create  comparable 
PLATO  and  non-PLATO  courses,  no  global 
comparisons,  l.e.,  previously  described 
dependent  variables,  measuring  PLATO 
versus  conventional  effects  would  be  made. 

No  attempts  would  be  made  to  compare 
training  times  of  lock-step  versus  self- 
paced  training. 

6)  ’^rimary  purpose  v;ould  be  to  use  PLATO  as 
a research  vehicle  directed  toward  use  of 
CAI/CMI  in  an  operational  training  setting. 

When  the  original  plans  for  the  service  test  were  abandoned, 
several  policy  questions  which  had  been  determined  before 
were  rendered  uncertain.  These  points  of  agreement  repre- 
sent an  attempt  to  reestablish  policy  as  well  as  to  enunci- 
ate intended  research  questions.  Also  the  first  of  these 
points  gives  TRAB  the  responsibility  of  publishing  a service 
test  plan. 

On  the  basis  of  these  agreements,  AFHRL  prepared  a 
draft  of  an  implementation  plan  and,  in  February,  1973,  sent 
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it  to  Chanute  for  comment.  The  plan  was  intended  as  an  out- 
line for  an  evaluation  plan  for  the  service  test.  Succeed- 
ing drafts  of  the  plan  were  to  make  it  applicable  to  the 
the  day-to-day  conduct  of  the  test  and  its  evaluation.  As 
with  the  previous  agreement  between  AFHRL  and  TRAB  person- 
nel, the  plan  was  motivated  by  the  concern  of  at  once  demon- 
strating the  PLATO  system  in  an  operational  setting  and 
using  the  PLATO  system  as  a vehicle  for  answering  research 
questions  regarding  CBE  methodologies  (Pennell  £t  £l.,  1973, 

p . 1 ) . 

The  plan's  reception  by  TRAB  was  less  than  enthusias- 
tic. In  the  view  of  TRAB,  the  plan  suffered  from  several 
deficiencies.  Among  these  were  the  fact  that  its  goals  were 
not  stated  in  measurable  terms,  that  the  plan  did  not 
clearly  set  out  the  division  of  labor  for  the  project,  that 
it  was  esoteric,  that  it  was  too  noncommittal  and  that  it 
was  general  where  it  should  have  been  specific  and  vice 
versa  (Main,  1973b). 

Third  plan.  To  settle  points  of  disagreement  and  fur- 
ther refine  the  project's  objectives,  another  meeting  was 
held  between  TRAB  and  the  AFHRL  project  monitor.  The  even- 
tual outcome  of  this  meeting  was  a draft  of  a new  plan. 

Both  TRAB  personnel  and  the  project  monitor  at  AFHRL  shared 
responsibility  for  writing  this  document.  After  some  minor 
revisions,  both  parties  approved  it  in  May,  1973,  nine 
months  into  the  project,  and  it  was  sent  to  ATC  for  final 
approval.  This  final  approval  was  never  granted.  ATC ' s 
reasons  for  withholding  approval  were  based  on  the  opinion 
that  the  plan's  objectives  were  beyond  the  capabilities  of 
the  service  test.  Perhaps  exhausted  from  the  effort  of 
writing  implementation  plans  for  the  service  test,  TRAB  used 
the  basic  ideas  of  this  final  attempt  to  guide  the  project 
until  a major  review  was  held  in  May,  1974.  It  was. 
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therefore,  effectively  the  project's  implementation  plan  for 
that  period  of  time  despite  the  absence  of  full  approval. 

This  plan  divided  the  project  into  three  stages  or 
phases.  The  first,  which  by  May,  1973i  bad  for  practical 
purposes  been  exhausted,  was  devoted  to  the  training  of  the 
PLATO  authors.  Experimentation  with  the  PLATO  system  would 
be  carried  out  during  the  second  phase.  During  the  final 
phase,  instructional  materials  developed  in  the  second  phase 
would  be  used  to  collect  cost  and  management  data.  Five 
areas  of  interest  were  designated  for  investigation: 
instructional  effectiveness  of  the  PLATO  system,  instructor/ 
programmer  characteristics,  human  factors  and  facility 
requirements,  methods  of  management  and  conduct  of  computer- 
based  training,  and  the  economics  of  using  computer-based 
instructional  materials  (Green  _et  al^.  , 1973,  p.  'O . 

Although  this  document  was  more  substantial  than  the 
second  plan,  much  of  that  weight  was  accounted  for  by  his- 
torical and  contextual  materials  included  in  the  plan. 

Many  of  the  same  criticisms  which  were  directed  towards  the 
preceding  plan  could  easily  be  applied  to  this  one.  In  the 
absence  of  any  precisely  defined  objectives,  it  was  directed 
at  using  the  PLATO  IV  system  as  a research  vehicle  to  gather 
in  as  many  areas  as  seemed  accessible  and  with  whatever 


methods  appeared  feasible.  This  attitude  towards  the  ser- 
vice test  is  expressed  clearly  in  the  introductory  discus- 
sion of  the  implementation  plan: 


The  following  sections  have  been  provided  to 
give  more  detail  on  the  specific  techniques 
and  methodologies  to  be  employed  in  accom- 
plishing the  service  test  goals.  A caveat  may 
be  appropriate  at  this  point,  however.  An 
exploratory  service  test  of  tfie  kind  planned 
for  tills  effort  must  be  highly  dynamic  to  pro- 
duce Its  most  useful  product.  Therefore, 
methods  and  procedures  detailed  below  must 
be  considered  mutable  and  subject  to  changes 
dictated  by  a heuristic  appt-oacii.  ( Ibid . , p.  9) 
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This  statement  reflects  accurately  the  lack  of  confidence 
the  plan's  authors  had  in  it.  Better  than  the  pages  spent 
in  articulating  the  guidelines  or,  as  experience  showed, 
hopes  for  the  service  test,  this  statement  characterizes 
the  activity  of  the  service  test  from  its  beginning  until 
May,  1974. 

As  stated  before,  ATC  never  gave  its  final  approval  to 
this  plan.  In  the  absence  of  any  other  plan,  however,  the 
managers  of  the  service  test  at  Chanute  AFB  used  it  as  a 
basis  for  administering  the  project  until  May,  1974.  At 
that  time  the  project  was  given  a full  review  which  resulted 
in  a drastic  change  in  direction.  Because  this  change  was 
so  radical  and  because  it  resulted  in  the  most  productive 
period  of  the  service  test,  the  change  and  its  consequences 
are  discussed  in  a separate  section  on  the  ISO  management. 
Here,  it  is  sufficient  to  note  that  after  the  May  1974 
review,  the  service  test's  goals  in  terms  of  products  and 
timelines  were  precisely  defined.  Moreover,  the  major 
effort  was  to  be  devoted  to  producing  operational  instruc- 
tional materials  on  the  PLATO  system  rather  than  using  PLATO 
lessons  individually  as  research  vehicles.  Research  under 
the  new  plan  was  to  be  confined  to  evaluation  of  the  effec- 
tiveness of  a computer-based  medium  in  an  operational 
setting . 

Inherent  Defects  In  The  Service  Test 

Before  turning  to  other  aspects  of  the  establishment  of 
Chanute's  CBE  Service  Test,  it  is  necessary  to  address  some 
comments  to  the  situation  which  effectively  impeded  the  suc- 
cessful writing  of  an  implementation  plan  for  almost  the 
first  half  of  the  project's  duration.  Interviews  with  ser- 
vice test  evaluators,  managers,  and  authors  along  with 
observation  of  the  test  throughout  its  duration  have 
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indicated  several  contributing  factors  to  this  circumstance. 
Those  which  were  inherent  in  the  service  test,  and  hence 
directly  affected  the  establishment,  will  be  dealt  with 
here.  Other  factors  will  be  treated  as  they  become  relevant 
to  the  succeeding  discussion. 

The  fundamental  source  for  indecisiveness  with  respect 
to  the  service  test's  objectives  can  be  found  in  the  differ- 
ent missions  of  the  two  Air  Force  agencies  responsible  for 
the  test,  AFHRL  and  ATC.  On  the  one  hand,  ATC  is  concerned 
with  the  technical  training  of  servicemen.  Its  interest, 
therefore,  was  to  discover  how  the  PLATO  system  would 
perform  as  a technical  training  medium.  AFHRL,  on  the 
other  hand,  with  its  responsibility  for  all  Air  Force 
behavioral  science  research  and  its  new  commitment  to  the 
Advanced  Instructional  System  (AIS),  a large  CBE  system 
under  development  at  Lowry  AFB,  was  naturally  concerned  to 
discover  as  much  as  possible  about  computer-based  media. ^ 
Thus,  ATC's  interests  in  the  PLATO  system  were  based  pri- 
marily in  its  capability  to  deliver  actual  instruction  and 
secondarily  with  the  potential  of  the  system  to  support 
basic  CEE  research;  priorities  of  interests  of  AFHRL  in 
these  two  areas  were  reversed  from  those  of  ATC. 

Practically,  the  interest  of  AFHRL  in  relatively  pure 
CBE  research  manifested  itself  in  dropping  the  first  imple- 
mentation plan.  If  implementing  the  service  test  as 
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originally  planned  in  the  GPVC  was  impossible,  surely  anoth- 
er course  could  have  been  selected  for  a similar  sort  of 
service  test.  According  to  personnel  involved  with  the  ser- 
vice test  at  that  time,  no  alternative  course  was  sought. 
ATC's  influence  and  prejudices  are  evident  by  the  fact  that 
it  withheld  approval  of  any  of  the  implementation  plans 
which  were  drawn  up  by  AFHRL. 

The  practical  effect  of  this  indecisiveness  for  the 
day-to-day  management  of  the  project  cannot  be  overesti- 
mated. Without  an  implemention  plan,  TRAB  personnel  who 
were  administering  the  test  at  Chanute  had  no  real  guide- 
lines with  respect  to  both  products  and  deadlines.  Inter- 
views with  members  of  Chanute  PLATO  project  indicate  that 
the  absence  of  clearly  enunciated,  fundamental  objectives 
and  the  consequent  lack  of  direction  from  those  agencies 
directly  responsible  for  the  service  test  severely  under- 
mined the  basis  of  authority  of  the  project's  local 
managers . 

In  addition  to  the  clash  of  fundamental  interests  of 
AFHRL  and  ATC  in  the  PLATO  service  test,  neither  of  these 
organizations  had  had  much  experience  with  computer-based 
education.  Moreover,  since  Chanute  was  the  first  military 
training  center  to  use  the  PLATO  system,  CERL  could  provide 
little  in  the  way  of  guidelines  or  advice  in  the  preparation 
times  for  technical  training  materials.  The  practical 
effect  of  this  lack  of  experience  was  that  none  of  the 
people  attempting  to  write  an  implementation  plan  had  a 
clear  concept  of  what  was  feasible  in  a three  year  PLATO 
service  test.  For  example,  being  novices  with  the  system, 
they  had  no  reasonable  estimate  of  how  many  manhours  per 
hour  of  student  instruction  would  be  needed  to  produce  even 
the  simplest  sorts  of  lessons.  Thus,  any  estimate  of  the 
magnitude  of  the  project's  total  output  was  purely  guess- 
work . 
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As  an  example  of  this  sort  of  guesswork,  consider  that 
when  the  project  began  it  was  proposed  to  use  the  PLATO 
system  to  duplicate  most  of  the  non-laboratory  instruction 
in  the  GPVC.  Thus,  it  was  hoped  that  about  173  hours  of 
instruction  would  be  produced.  As  far  as  is  known  no  consi- 
deration was  given  at  that  time  to  the  work  load  that  such 
an  objective  would  place  on  the  PLATO  authors.  In  any  case, 
less  than  six  months  later  the  objective  for  the  service 
test  was  reduced  to  80  hours  of  CBE  lesson  material.  This 
objective  was  essentially  halved  again  in  May,  1973,  when 
TRAB  and  AFHRL  made  their  final  attempt  to  write  a joint 
implementation  plan.  This  last  estimate  is  unique  among  its 
predecessors  in  that  it  is  based  on  an  estimated  rate  of 
lesson  production,  namely,  that  observed  in  the  early  por- 
tions of  the  service  test.  If  objectives  about  particulars 

I 

: such  as  the  quantity  of  lesson  materials  to  be  produced  were 

so  casually  made,  it  is  easy  to  see  how  the  other  objectives 
for  which  there  was  no  ground  for  making  production  esti- 
mates would  probably  be  more  unrealistic. 

Philosophical  interlude.  In  view  of  this  naiveti  with 
, respect  to  CBE,  the  TRAB  managers  of  the  service  test  should 

not  have  been  given  the  license  by  the  AFHRL  project  moni- 

► 

tors  and  evaluators  of  being  able  to  change  the  project's 
objectives  in  the  face  of  day-to-day  contingencies.  Such  a 
policy  would  be  most  likely  to  lead  to  so  many  changes  in 
direction  that  at  its  conclusion  the  project  would  have  made 
' very  little  progress  in  any  given  direction.  Given  this 

inexperience,  the  project  planners  would  have  been  best 
advised  to  follow  the  recommendation  of  Ren^  Descartes  on 
I*  how  to  embark  on  projects  in  which  the  way  to  proceed  is 

less  than  clear.  In  such  cases,  Descartes  says  that  one 
must  early  establish  some  rule  for  action  and  resolutely 
follow  it.  Even  if  the  rule  is  uncertain  and  if  it  is  known 
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to  be  doubtful,  it  must  be  treated  as  certain.  In  this  way, 
one  would  be 

f following  the  example  of  travelers,  who,  finding 

themselves  lost  in  a forest,  know  that  they  ought 
not  to  wander  first  to  one  side  and  then  to  the 
other,  nor,  still  less,  to  stop  in  one  place,  but 
understand  that  they  should  continue  to  walk  as 
' straight  as  they  can  in  one  direction,  not  diverg- 

ing for  any  slight  reason,  even  though  it  was 
; possibly  chance  alone  that  first  determined  them  in 

! their  choice.  By  this  means  if  they  do  not  go 

exactly  where  they  wish,  they  will  at  last  arrive 
' somewhere  in  the  end,  where  probably  they  will  be 

better  off  than  in  the  middle  of  a forest.  And 
thus  since  often  enough  in  the  actions  of  life  no 
’ delay  is  permissible,  it  is  very  certain  that,  when 

it  is  beyond  our  power  to  discern  the  opinions 
which  carry  the  most  truth,  we  should  follow  the 
most  probable;  and  even  although  we  notice  no 
greater  probability  in  one  opinion  than  in  the 
other,  we  at  least  should  make  up  our  minds  to 
f follow  a particular  one  and  afterwards  consider  it 

as  no  longer  doubtful  in  its  relationship  to 
practice,  but  as  very  true  and  very  certain, 
inasmuch  as  the  reason  which  caused  us  to  deter- 
mine upon  it  is  known  to  be  so.  (Descartes,  p.  96) 


The  sagacity  of  Descartes'  advice  was  not  called  into 
doubt  by  the  Chanute  service  test,  which  changed  implemen- 
tation plans  at  least  three  times  during  the  first  half  of 
its  intended  duration  and  never  finally  fixed  on  any  one 
plan.  Before  a final  management  plan  was  adopted,  the  major 
accomplishment  of  the  service  test  consisted  in  the  training 
of  the  authors.  Although  the  lessons  they  developed  did 
serve  as  a basis  during  the  second  phase  of  the  project, 
these  lessons  had  neither  the  cohesiveness  nor  polish  to  be 
implemented  as  bona  fide  CBE  lessons.  The  effect  of 
changing  plans  so  often  was  that  any  progress  made  during 
one  period  was  canceled  out  by  progress  in  another. 
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Selection  and  Training  of  PLATO  Authors 

The  selection  and  training  of  the  PLATO  authors  was  not 
delayed  or  substantially  impaired  by  the  absence  of  an 
implementation  plan.  By  early  August,  1972,  four  civilian 
and  four  military  authors  had  been  identified  by  the  mana- 
gers of  the  service  test  and  were  ready  to  begin  training. 
These  eight  authors  were  originally  put  under  the  supervi- 
sion of  the  chief  of  the  GPVC  along  with  31  other  personnel 
authorized  as  instructors  in  the  course.  The  fact  that  the 
authors  were  under  the  management  of  the  GPVC  chief  and  of 
TRAB  personnel  simultaneously  was  the  source  of  much  diffi- 
culty in  the  early  administration  of  the  service  test. 

The  branch  chief  of  TRAB  who  wrote  the  original  RTR  was 
responsible  also  for  selecting  authors  for  the  PLATO  service 
test.  Youth  was  one  of  the  primary  qualities  he  sought  in 
potential  PLATO  authors.  He  felt  that  relatively  young 
instructors  would  be  sufficiently  adaptable  to  revise  their 
educational  outlooks  and  work  habits  to  suit  the  needs  of 
using  the  PLATO  system  as  a teaching  medium.  He  hypothe- 
sized that  not  only  would  the  authors  have  to  use  new 
instructional  techniques  but  that  they  would  also  have  to 
master  a computer  programming  language  and  the  protocols  of 
working  with  a computer  system. 

The  TRAB  branch  chief  employed  two  other  criteria  in 
selecting  the  project's  authors.  These  two  criteria  were 
amount  of  education  and  willingness  to  stay  with  the  PLATO 
service  test  throughout  its  duration.  Selection  of  the  most 
highly  educated  personnel  available,  in  the  opinion  of  the 
branch  chief,  would  increase  the  likelihood  that  the  authors 
would  possess  the  intellectual  tools  and  discipline  to  mas- 
ter the  complex  skills  needed  to  author  materials  on  the 
PLATO  system.  Learning  these  skills,  moreover,  involved  so 
much  training  that  the  loss  of  an  author  during  the  service 


of  the  time  needed  to  train  a replacement. 

The  criterion  that  the  authors  be  subject  matter 
experts  in  the  intended  areas  of  CBE  lesson  development  was 
not  of  high  priority  in  author  selection.  Since  the  pool  of 


instructors  from  which  the  authors  were  selected  consisted 
of  GPVC  instructors,  it  was  assumed  that  any  of  the  candi- 
dates would  be  highly  knowledgeable  in  the  subject  matter  of 
that  course.  The  fact  that  a premium  was  placed  on  youth, 
however,  resulted  in  the  selection  of  some  authors  who  did 
not  have  much  experience  with  teaching  or  teaching  the 
course's  subject  matter. 

It  is  not  known  whether  experience  in  the  areas  of 
instructional  design  and  computer  science  was  considered  an 
important  criterion  in  author  selection,  but,  because  of 
available  candidates,  could  not  be  satisfied,  or  whether 
these  characteristics  were  not  considered  important.  In  any 
case.  Table  1 shows  that  the  authors  selected  collectively 
possessed  no  experience  in  the  first  of  these  areas  and  only 
a little  in  the  second.  Finally,  the  TRAB  branch  chief  was 
constrained  to  select  four  civilian  and  four  military  per- 
sonnel for  the  PLATO  author  staff.  Table  1 summarizes  the 
characteristics  of  the  staff. 

Table  1 shows  that  some  of  the  primary  selection  cri- 
teria were  well  met  by  the  authoring  staff.  With  respect 
to  getting  staff  members  to  stay  for  the  duration  of  the 
project,  however,  there  was  much  less  success.  Two  of  the 
military  authors  left  the  Air  Force,  and  hence  the  project, 
after  the  project  was  little  more  than  a year  old.  Another 
military  author  left  the  project  after  18  months.  Also  one 
civilian  author  left  the  project  after  two  years.  Because 
of  a scarcity  of  qualified  personnel  and  bureaucratic 
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Characteristics  of  the  Chanute  PLATO  Authors  in  July,  1972 


made  for  those  authors  who  departed  from  the  project  before 
the  intended  completion  date  of  July,  1975. 

Acquisition  of  PLATO  IV  term.inals  and  Peripheral  Devices 

The  Chanute  PLATO  service  test  was  allocated  a total  of 
25  PLATO  terminals.  The  delivery  of  these  terminals  was  to 
be  completed  by  the  end  of  the  second  year  of  the  service 
test.  Each  terminal  was  to  be  equipped  with  a random  access 
slide  selector  (Francis,  1976,  p.  ^),  a random  access  audio 
playback  device  (Ibid . , 1976,  p.  38),  and  a touch  panel 
(Ibid . , 1976,  p.  25). 

Because  of  manufacturers'  delays,  CERL  repeatedly 
revised  times  the  schedules  for  the  delivery  of  terminals 
and  peripherals.  The  inability  to  follow  any  of  these 
schedules  stems  primarily  from,  delays  in  delivery  of  hard- 
ware to  CERL  from  various  manufacturers.  The  delays  of  the 
manufacturers  on  the  other  hand  may  be  attributed  to  the 
fact  that  much  of  the  hardware  for  the  new  PLATO  IV  system 
was  being  manufactured  for  the  first  time  so  that  these 
manufacturers  were  encountering  the  problems  that  usually 
accompany  the  assembly  of  new  products.  For  Chanute,  the 
source  of  these  delays  was  not  important.  All  that  con- 
cerned this  site  was  that  the  delays  did  exist,  that  they 
were  long,  and  that  they  impeded  the  ability  of  the  newly 
trained  PLATO  authors  to  develop  their  skills  in  using  the 
PLATO  system. 

Chanute’s  first  two  terminals  arrived  two  weeks  late  in 
September,  1972.  This  delay  forced  the  authors  to  travel 
daily  to  the  University  of  Illinois  for  two  weeks  to  con- 
tinue to  use  the  PLATO  IV  terminals  there  (see  Appendix  1). 
The  authors  had  just  completed  their  TUTOR  training  at  that 
time.  Consequently  the  importance  of  getting  extensive 


exposure  to  the  PLATO  system  far  outweighed  the  inconve- 
nience of  working  in  an  unfamiliar  and  temporary  environment 
and  of  commuting  between  Chanute  AFB  and  CERL,  a trip  of  18 
miles  each  way. 

Another  ill  effect  of  the  slow  delivery  of  PLATO  ter- 
minals was  that  the  eight  Chanute  authors  were  split  into 
three  groups  with  each  group  working  one  of  the  three  work- 
day shifts.  (This  was  possible  since  PLATO  system  service 
was  available  22  hours  a day.)  As  the  shifts  were  originally 
established  and  remained  through  June,  1973,  two  shifts  each 
with  three  authors  did  not  interact  at  all.  Not  only  did 
this  arrangement  hinder  communication  within  the  authoring 
group,  it  also  often  rendered  ineffectual  the  attempts  of 
the  TRAB  personnel  to  monitor  and  guide  the  progress  of  the 
authors.  In  June,  1973,  the  shifts  were  differently  stag- 
gered to  avoid  some  of  these  problems.  It  was  not  until 
January,  1974,  when  the  eighth  terminal  was  installed  that 
all  of  the  authors  could  work  the  same  shift. 

The  slow  delivery  of  terminals  was  perceived  by  the  on- 
site managers  as  one  of  the  project's  biggest  problems 
during  its  initial  stages; 

In  the  beginning  of  a project,  one  additional  ter- 
minal can  make  all  the  difference  as  far  as  avail- 
able terminal  hours  and  flexibility  of  scheduling. 

Never  knowing  when  the  next  terminal  would  be 
delivered  and  always  being  promised  that  it  would 
be  there  the  next  week  added  more  to  the  list  of 
frustrations.  (Green,  1973,  p.  11) 

The  inability  of  CERL  to  deliver  Chanute's  terminals 
according  to  most  of  the  proposed  schedules  also  damaged  the 
confidence  of  the  project  adminstrator s in  CERL's  promises 
for  delivery  of  other  hardware.  In  the  case  of  the  touch 
panels,  the  lack  of  confidence  had  a definite  impact  on 
courseware  development.  Although  enough  panels  were 
installed  for  author  use,  the  project  administrators  refused 
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to  allow  their  authors  to  incorporate  use  of  the  touch  panel 
into  the  PLATO  lessons  even  though  delivery  of  a sufficient 
number  was  promised  by  CERL  before  the  students  would  be 
using  them.  Only  with  all  of  the  touch  panels  installed 
would  such  reliance  on  them  in  the  PLATO  lessons  be  per- 
mitted. For  this  reason,  touch  panels  were  not  used  in  any 
of  Chanute’s  operational  lessons  although  each  terminal  was 
eventually  accoutered  with  one. 

The  situation  with  audio  devices  was  much  the  same  as 
with  touch  panels.  Since  the  site  was  not  completely  sup- 
plied with  these  peripherals,  it  was  decided  not  to  use  them 
in  the  CBE  lessons.  In  fact,  for  the  entire  duration  of  the 
service  test  and  the  subsequent  operational  use  of  the  PLATO 
lessons,  there  was  never  more  than  one  audio  device  at  Cha- 
nute,  and  it  was  seldom,  if  ever,  used. 

With  respect  to  the  slide  selectors  the  case  was  some- 
what more  fortunate.  A full  complement  of  these  was 
installed  early  enough  so  that  the  decision  to  use  micro- 
fiche in  the  CBE  lessons  could  be  made.  The  slide  selec- 
tors, nevertheless,  required  frequent  on-site  maintenance 
and  were  not  found  to  be  reliable  by  the  Chanute  authors 
(Francis,  1976,  p.  30).  An  extensive  description  of  Cha- 
nute's  experience  in  preparing  microfiche  for  their  lessons 
may  be  found  on  pages  19-32  of  Francis  (1976). 

Service  Test  Managers 

The  multiplicity  of  interests  in  the  Chanute  PLATO  IV 
service  test  has  already  been  detailed  in  the  discussion  of 
the  research  agreement.  It  should  be  recalled  that  the 
PLATO  authors  were  officially  designated  as  instructors  in 
the  GPVC.  At  the  same  time,  the  entire  PLATO  service  test 
was^ under  the  general  guidance  of  AFHRL  which  was  trying  to 
exeVcise  its  supervisory  duties  from  a distance  of  900  miles 
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from  the  test  site.  Finally,  TRAB  personnel  located  at  Cha- 
nute  AFB  acted  as  on-site  project  managers  sometimes  expe- 
diting directives  from  AFHRL,  sometimes  collaborating  with 
AFHRL  in  formulating  project  plans,  and  sometimes  taking  the 
initiative  in  administering  and  directing  the  service  test. 
All  in  all,  it  appears  that  there  was  an  attempt  to  compen- 
sate for  the  absence  of  an  implementation  plan  with  a sur- 
feit of  project  administrators. 

Of  the  three  organizations  directly  involved  with  the 
service  test,  TRAB  was  by  far  the  most  important.  The  ori- 
ginal Research  Agreement  designated  TRAB  as  the 

Technical  School  OPR  [Office  of  Primary 
Responsibility]  for  coordination  of  the 
experimental  service  test  with  ATC/XPT , AFHRL, 

ARPA,  and  the  University  of  Illinois.  This 
office  will  monitor  the  research  program  by 
developing  a programmed  action  directive  for  the 
duration  of  the  three-year  research  agreement . 

(Training  Research  Agreement,  1972) 


What  monitoring  a research  program  by  means  of  a "programmed 
action  directive”  may  mean  is  not  altogether  clear.  How- 
ever, the  TRAB  Branch  Chief  and  his  staff  members  were 
de  facto  project  officers  of  the  service  test  from  its 
inception  until  May,  197^. 

By  "project  officers”  something  less  than  might  ordi- 
narily be  understood  is  meant,  for  in  any  effort  to  direct 
the  project,  TRAB  personnel  had  to  contend  with  the  fact 
i that  the  PLATO  authors  were  under  the  supervision  of  the 

GPVC  branch  chief.  Without  the  cooperation  of  the  GPVC 
chief,  therefore,  the  influence  of  the  TRAB  personnel  was 


limited.  Again,  without  some  convergence  of  goals  of  ATC 
and  AFHRL,  the  administration  of  the  project  on-site  was 
indecisive  at  best.  An  awareness  by  the  TRAB  personnel  of 


diversity 

following 


of  influences  affecting  TRAB  is  apparent  in 
quote  from  a documentation  of  early  project 
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problems : 

It  seems  apparent  that  TTOE-^  personnel  are  the 
critical  link  among  all  contacts  concerned  with 
the  PLATO  effort.  Everyone  involved  with  the 
program  at  whatever  level  seems  to  have  their 
own  interests  uppermost  in  mind  when  any  problem 
about  PLATO  is  brought  to  their  attention. 

(Green,  1972,  p.  4) 

Clearly,  since  its  role  in  the  service  test  was  subor- 
dinate to  that  of  AFHRL  and  ATC,  TRAB  could  not  administer 
the  project  effectively  without  some  convergence  of  inter- 
ests of  those  two  agencies  and  without  an  approved  implemen- 
tation plan.  There  were  other  conditions  that  together  made 
effective  leadership  of  the  project  beyond  the  practical 
capabilities  of  TRAB.  Among  these  conditions  were  frequent 
personnel  changes  in  TRAB  and,  consequently,  in  the  PLATO 
project's  administrator  during  the  first  two  years  of  the 
test.  These  changes  along  with  an  assessment  of  their 
impact  on  the  project  is  contained  in  the  section  discussing 
management  in  the  first  two  years  of  the  project. 

Since  the  leadership  of  the  project  changed  so  often  in 
its  first  two  years  it  would  be  difficult  to  categorize  the 
qualifications  of  the  TRAB  personnel.  In  interviews  with 
members  of  TRAB's  staff,  however,  one  deficiency  in  particu- 
lar was  emphasized  as  being  especially  detrimental  to  the 
success  of  the  test.  During  its  first  two  years,  TRAB  nei- 
ther had  any  staff  members  experienced  in  curriculum  design 
nor  ready  access  to  Air  Force  personnel  with  experience  in 
this  field.  This  deficiency  was  characterized  by  a member 
of  the  TRAB  staff  who  had  held  a position  of  high  responsi- 
bility in  the  administration  of  the  project  as  being  the 
single  most  important  factor  in  contributing  to  the  relative 
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lack  of  progress  in  the  project's  first  two  years.  He  made 
this  assessment  in  comparing  this  problem  with  those  of 
trying  to  develop  an  implementation  plan  that  would  satisfy 
both  AFHRL  and  ATC  and  of  trying  to  undertake  an  ambitious 
curriculum  development  project  on  a newly  assembled  innova- 
tive CBE  system. 

m 

It  should  be  recalled  that  none  of  the  PLATO  authors 
had  a significant  background  in  instructional  design  and 
that  none  of  them  had  any  background  in  developing  lessons 
for  CBE.  It  is  then  easy  to  see  that  with  this  naivety  with 

i 

respect  to  CBE  instructional  design  on  the  part  of  the 
authoring  staff,  the  absence  of  experience  in  curriculum 
design  could  be  critical  to  the  project's  outcome.  It  is 
merely  speculation  to  attribute  some  of  the  problems  of  the 
project  to  the  absence  of  a curriculum  designer.  At  another 
ARPA/PLATO  site  which  was  administered  by  an  individual  with 
ten  years  of  curriculum  design  experience,  fewer  of  these 
problems  arose  (Himwich,  1977)*  This  relative  competence 
was  not  only  apparent  in  daily  management  of  project  affairs 
but  also  in  allocating  responsibilities  to  the  staff's 
members  so  that  the  project's  long  range  goals  could  be 

achieved.  ^ 

Summary  I 

The  Air  Force's  PLATO  service  test  had  several  diffi- 
culties which  were  either  inherent  in  the  structure  of  the 
test  or  could  be  attributable  to  causes  beyond  the  control 
of  the  project's  planners  and  administrators.  Since  these 
problems  were  the  causes  of  several  deficiencies  in  the 
establishment  of  the  test  and  because  they  continued  to 
plague  the  project  throughout  its  duration,  it  is  fitting  by 
way  of  summary  of  the  project  establishment  to  enumerate 
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them  here. 

The  problems  that  arose  in  the  service  test  seem  to  be 
from  three  fundamental  sources: 

1)  The  two  Air  Force  agencies — ATC  and  AFHRL — 
which  had  primary  responsibilities  for  the  ser- 
vice test  had  different  aspirations  for  the 
outcome  of  the  project.  ATC  was  interested  in 
PLATO  mainly  as  an  operational  teaching  device 
while  AFHRL  tended  to  view  the  PLATO  system 

as  a vehicle  of  basic  CBE  research. 

2)  The  service  test  began  at  the  same  time  that 
the  new  PLATO  IV  system  was  being  assembled. 

3)  Neither  among  the  supervisors  of  the  project 
nor  among  the  PLATO  authors  was  there  any  signi- 
ficant experience  in  curriculum  and  instruc- 
tional design. 

The  first  of  these  has  been  discussed  as  a primary 
reason  that  no  implementation  plan  was  officially  adopted 
for  the  project.  Also,  as  will  be  more  apparent  in  suc- 
ceeding sections  of  this  chapter,  it  made  effective  admini- 
stration of  the  project  at  Chanute  AFB  by  TRAB  difficult. 
However,  it  is  clear  that  the  task  of  managing  the  project 
locally  and  simultaneously  serving  the  interests  of  ATC  and 
AFHRL  could  only  be  successfully  completed  with  the  aid  of 
exceptional  administrative  talent  or  of  exceptionally  good 
fortune.  Chanute  was  blessed  with  neither. 

One  of  the  manifestations  of  the  second  of  these  pro- 
blems appeared  in  the  delays  in  the  delivery  of  terminals 
and  peripheral  hardware.  These  delays,  in  turn,  were  an 
impediment  to  the  project  authors  in  practicing  skills  they 
had  just  been  taught  in  author  training  at  CERL.  Finally, 
with  the  start-up  of  a computer  system  as  complex  and  large 
as  the  PLATO  IV  system  some  difficulties  with  regard  to 
establishing  reliability  must  be  expected.  Being  the  first 
ARPA/PLATO  site  and  one  of  the  first  large  sites  remote  from 
the  University  of  Illinois,  the  Chanute  service  test  fell 
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victim  to  a number  of  problems  that  would  not  and  did  not 
trouble  later  sites.  These  problems  will  be  discussed 
wherever  appropriate  in  succeeding  sections  of  this  chapter 
The  absence  of  experience  in  curriculum  and  instruc- 
tional design  in  the  project  had  its  main  impact  in  the 
manner  in  which  the  project  authors  were  administered.  It 
is,  therefore,  sufficient  to  emphasize  this  fact  here  and 
discuss  it  in  the  section  on  project  management. 
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SERVICE  TEST  MANGEMENT  BEFORE  MAY,  197M 

The  management  of  Chanute's  PLATO  service  test  under- 
went several  changes  in  the  first  portion  of  the  project. 

These  changes  are  in  part  attributable  to  turnovers  in  the 
staffing  of  TRAB  which  had  the  responsibility  of  locally  ; 

monrtoring  the  project  and  of  coordinating  the  efforts  of 
the  agencies  that  were  involved  with  the  project.  Other 
changes  occurred  as  some  workable  scheme  for  administrating 
the  daily  progress  of  the  authoring  staff  was  sought.  These 
changes  are  important  because  they  revealed  defects  in  the 
original  structure  of  the  service  test  and  because  they  had 
an  effect  on  the  authoring  staff.  The  first  part  of  this 
section  discusses  the  evolution  of  the  service  test  manage- 
ment. The  second  part  will  note  other  influences  on  the 
management  of  the  project.  Finally,  the  lesson  development 
processes  during  this  phase  of  the  service  test  will  be 
described . 

Evolution  of  the  Project  Management  before  May,  1974 

While  the  leadership  of  TRAB  could  not  be  well  directed 
without  a convergence  of  interests  of  ATC  and  AFHRL  and 
without  an  approved  implementation  plan,  the  fact  that  TRAB 
underwent  a number  of  personnel  changes  during  the  early 
stages  of  the  project  also  contributed  to  TRAP'S  indecisive 
leadership  at  the  site  level.  Which,  if  any,  of  these 
changes  were  most  detrimental  is  a matter  for  speculation. 

However,  it  can  not  be  doubted  that,  taken  separately  or 
together,  they  did  have  adverse  influence  on  the  project's 
outcome . 

The  first  change  in  local  leadership  of  the  project 
occurred  shortly  after  the  authors  had  completed  their  PLATO 
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training  at  CERL  in  September,  1972.  At  that  time,  the  TRAB 
branch  chief  left  Chanute  AFB.  This  individual,  by  virtue 
of  his  proximity  to  the  University  of  Illinois,  had  been 
aware  of  the  development  of  the  PLATO  IV  system  at  CERL. 
Being  excited  by  the  potential  of  a powerful  computer-based 
educational  system,  he  initiated  the  project's  RTR . Also, 
the  central  idea  of  evaluating  the  effectiveness  of  the 
PL^"^0  IV  system  by  duplicating  the  already  existing  GPVC  was 
his.  It  is  not  remarkable  that  enthusiasm  for  this  approach 
to  the  service  test  diminished  after  his  departure  in  the 
face  of  other  difficulties  to  that  concept's  implementation. 

At  the  time  of  the  departure  of  the  first  project  offi- 
cer, one  of  the  psychologists  from  TRAB's  military  personnel 
was  assigned  as  project  officer  working  one-half  time  on  the 
PLATO  project.  TRAB's  office  and  that  of  the  PLATO  authors 
were  not  of  sufficient  proximity  to  each  another  for  the  new 
project  officer  to  get  a satisfactory  feeling  for  the  needs 
of  the  project.  To  remedy  this  situation,  he  obtained  a 
desk  within  the  PLATO  group.  Once  he  began  to  have  more 
contact  with  the  PLATO  authors,  however,  his  involvement 
with  the  project  increased.  By  January,  1973,  he  was  work- 
ing full-time  with  the  project  (Green,  1973,  p.  8). 

Increased  commitment  to  the  PLATO  project  of  all  mem- 
bers of  TRAB  followed  that  of  the  project  officer.  TRAB's 
staff  consisted  of  two  military  psychologists,  a civilian 
training  specialist,  and  a civilian  secretary.  One  of  the 
psychologists  was  already  acting  as  project  officer.  By 
early  April,  the  other  psychologist  and  the  training  spe- 
cialist were  spending  a significant  portion  of  their  time 
with  the  PLATO  project  also.  At  that  time,  in  fact,  serious 
consideration  was  given  to  the  idea  of  moving  the  entire 
TRAB  office  to  a room  adjoining  the  PLATO  section  (Green, 
1973,  p.  8).  . ‘ 
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While  TRAB's  involvement  was  increasing,  some  difficul- 
ties between  the  PLATO  group  and  the  other  instructors  from 
the  GPVC  arose.  From  the  outset,  the  PLATO  staff,  though 
part  of  the  GPVC  staff,  was  viewed  with  some  envy  by  their 
colleagues  who  taught  the  conventional  course.  It  was  felt 
that  since  the  PLATO  staff  did  not  have  to  instruct  as  con- 
ventional lecturers  that  their  situation  was  preferable  to 
that  of  the  other  instructors.  Moreover,  early  in  the  pro- 
ject there  were  only  two  terminals  installed  at  Chanute. 
Thus,  only  two  authors  could  use  the  PLATO  system  at  a time 
while  the  other  six  were  left  to  plan  their  lessons  at  their 
desks.  This  behavior  was  judged  to  be  non-productive  by  the 
conventional  instructors  and  the  GPVC  supervisory  personnel. 

Early  in  1973,  because  of  hardware  and  software  acqui- 
sition and  modification,  the  PLATO  system  was  unavailable 
for  six  weeks.  Restricted  entirely  to  desk  work,  the  labors 
of  the  CBE  authors  seemed  to  be  even  more  unproductive  than 
they  had  appeared  earlier.  This  appearance,  it  was  argued, 
had  a demoralizing  effect  on  the  conventional  instructors. 

To  remedy  the  apparent  inequity,  the  PLATO  authors,  military 
and  civilian  alike,  were  ordered  to  take  part  in  custodial 
duties  that  required  several  hours  a day  of  stripping, 
cleaning,  and  waxing  floors.  These  duties  were  even  main- 
tained after  the  PLATO  system  was  once  again  available  on  a 
reliable  basis.  In  fact,  it  was  only  after  hard  and  long 
argumentation  from  the  TRAB  project  officer  that  the  GPVC 
Branch  Chief  was  convinced  that  the  PLATO  staff  had  enough 
to  do  without  these  additional , duties . 

The  friction  between  the  PLATO  authors  and  the  other 
GPVC  instructors  was  heightened  by  the  ambiguity  in  the 
leadership  of  the  project.  On  the  one  hand,  the  GPVC  chief 
was  officially  the  supervisor  of  the  CBE  authors,  while  on 
the  other  TRAB  personnel  had  taken  the  active  role  in 
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administering  the  project.  Again,  the  GPVC  chief  "knew 
little,  if  anything,  about  the  project"  (Green,  1973,  p.  8) 
while  members  of  TRAB  had  taken  the  time  and  trouble  to 
learn  TUTOR,  the  programming  language  used  on  the  PLATO 
system,  as  well  as  other  efforts  to  help  the  authors  in 
undertaking  the  task  of  developing  CBE  materials.  Thus 
TRAB,  with  no  official  sanction  to  supervise  the  CBE 
authors,  was  in  fact  doing  so  while  the  officially  desig- 
nated supervisors  were  taking  very  little  interest  in  any 
aspect  of  the  service  test. 

In  early  1973,  an  attempt  was  made  to  alleviate  the 
the  ambiguity  in  leadership  of  the  PLATO  projects  and  to 
reduce  somewhat  the  friction  that  had  developed  between  the 
PLATO  authors,  as  GPVC  instructors,  and  the  other  course 
instructors  teaching  the  conventional  GPVC.  To  solve  these 
problems,  discussions  were  begun  concerning  the  feasibility 
of  transferring  the  authors  to  TRAB.  This  transfer  was  in 
fact  made  with  the  understanding  that  the  PLATO  authors 
would  revert  to  the  supervision  of  the  GPVC  chief  once  their 
materials  were  completed  and  were  being  used  operationally. 
With  this  change  in  the  management  of  the  project  and  the 
involvement  of  the  entire  TRAB,  the  authors  were  under  a 
single  local  leadership. 

When  TRAB  began  administering  the  daily  affairs  of  the 
service  test,  the  need  for  a larger  research  component  for 
the  project  became  apparent  to  TRAB  personnel  (Green  ^ , 

1973,  p.  40).  To  assist  in  monitoring  this  need,  TRAB  urged 
AFHRL  to  meet  the  terms  of  the  original  agreement  between 
ATC  and  AFHRL  and  provide  an  on-site  AFHRL  representative 
(Green,  1973,  p.  9).  In  June  1973,  AFHRL  hired  an  indivi- 
dual to  serve  in  this  role.  The  view  of  TRAB  towards  this 
appointment  was  quite  sanguine.  TRAB  anticipated  that  "This 
action  will  insure  that  research  needs  under  jurisdiction  of 


Air  Training  Command  are  properly  balanced."  (Green  ^ al . , 
1973,  p.  40)^.  An  immediate  benefit  of  the  appointment  was 
that  communications  between  the  service  test  site  and  AFHRL 
at  Lowry  AFB,  Colorado,  were  improved. 

For  service  test  monitor  and  liaison,  AFHRL  selected 
an  individual  who  had  been  an  instructor  at  Chanute  AFB  for 
five  years.  As  an  instructor  he  had  probably  gained  some 
experience  of  with  instructional  design  and  implementation 
of  instruction.  However,  there  is  little  in  this  background 
that  would  indicate  a competence  in  evaluation,  management, 
or  curriculum  design.  With  experience  only  in  the  area  of 
instruction,  this  individual  was  not  prepared  to  insure  that 
the  service  test  maintained  a "proper  balance"  between 
research  and  operational  concerns. 

The  management  role  of  the  AFHRL  monitor  was  very 
important  during  the  first  phase  of  the  service  test. 

Despite  his  inexperience  in  the  important  areas  of  educa- 
tional research,  curriculum  and  instructional  design,  and 
management,  he  and  the  TRAB  project  officer  were  very  nearly 
peers  in  these  areas.  Thus,  his  opinions,  which  were 
already  invested  with  the  weight  of  his  position  as  AFHRL's 
repr esentive , generally  influenced  the  project  officer's 
decisions  strongly.  Together  the  AFHRL  representative  and 
TRAB  project  officer  administered  the  service  test  from  the 
time  the  former  assumed  his  duties  in  June,  1973,  until  the 
first  phase  of  the  project  was  formally  reviewed  in  May, 

^This  statement  is  of  significance  not  only  as  documen- 
tation of  what  role  the  AFHRL  liaison  was  expected  to 
fill  but  as  an  indication  that  TRAB  was  aware  of  the 
difficulties  in  trying  to  administer  a project  that  was 
to  be  simultaneously  research  and  operationally  oriented. 
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1974.  For  this  reason,  the  administration  of  the  project 
during  this  period  may  be  justifiably  attributed  to  the 
"management/evaluation"  staff. 

During  the  tenure  of  this  management/evaluation  staff, 
the  authors  essentially  operated  autonomously.  For  most  of 
this  time,  the  project  only  had  four  PLATO  terminals.  To 
maximize  the  time  that  each  author  could  work  on  the  PLATO 
system,  the  staff  worked  three  daily  shifts.  For  the 
authors,  working  under  such  conditions  decreased  somewhat 
group  interactions  that  are  useful  in  solving  problems  that 
arise  in  lesson  development  and  in  utilizing  the  variety  of 
talents  in  an  authoring  group.  For  the  management/evalua- 
tion staff,  splitting  the  authors  into  three  shifts  meant 
that  the  authors  often  worked  without  supervision  and  that 
their  progress  was  difficult  to  monitor. 

Having  established  the  habit  of  working  autonomously 
the  authors  naturally  resisted  attempts  to  establish  any 
regimentation  in  their  work  habits.  This  resistance 
amounted  to  more  than  the  mere  distaste  for  supervision.  On 
one  occasion,  when  the  project  was  well  into  its  second 
year,  the  AFHRL  monitor  proposed  an  implementation  plan.  A 
strong  negative  reaction  to  this  plan  by  the  service  test 
authors  was  sufficient  to  prevent  him  from  formally  propos- 
ing the  plan  to  AFHRL  and  ATC.  At  other  times,  the  authors, 
both  individually  and  in  groups,  undertook  to  reform  the 
project  themselves.  With  respect  to  providing  some  direc- 
tion to  their  work,  their  efforts  were  as  ineffectual  as 
those  of  the  management/evaluation  staff.  It  is,  however, 
noteworthy  that  they  undertook  such  initiatives  while 
occupying  subordinate  positions.  It  is  a token  of  the  lack 
of  direction  and  order  that  pervaded  the  project  during  this 
phase . 
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With  the  possible  exception  of  the  first  writer  of  the 
RTR,  the  succession  of  TRAB  project  officers  and  the  on-site 
AFHRL  representative  were  all  actively  engaged  in  graduate 
degree  programs  at  the  University  of  Illinois.  Two  of  the  ] 

TRAB  project  officers  were  involved  in  doctoral  programs.  | 

One  of  these  officers  felt  so  acutely  the  demands  of  simul-  | 

taneously  managing  a CBE  curriculum  development  project  and  i 

of  pursuing  an  advanced  degree  that  he  left  the  Air  Force  to  1 

become  a full-time  graduate  student.  Another  took  leave  of 
his  Air  Force  duties  to  devote  his  energies  full-time  to  his 
graduate  program. 

It  is  impossible  to  assess  the  overall  impact  that  \ 

enrollment  in  graduate  degree  programs  had  on  the  quality  of 
the  management  of  the  Chanute  service  test.  Enhanced  back- 
grounds in  education  and  evaluation  were  valuable  to  the 
management  of  the  project.  However,  the  courses  and  studies 
were  directed  at  long-term  career  objectives  and  so  were 
more  extensive  than  mere  supplementation  of  knowledge  needed  i 

to  direct  the  service  test.  Also,  in  two  cases  the  commit-  ; 

ments  of  a project  officer  were  radically  altered  for  the  i 

sake  of  continuing  graduate  studies.  It  is,  therefore,  j 

plausible  to  conclude  that  their  studies  on  the  whole  j 

I 

detracted  from  the  performance  of  their  duties  as  project  i 

managers.  How  much  these  duties  were  affected  can  only  be  a 
matter  of  speculation.  j 

From  interviews  with  service  test  authors,  it  has  been  j 

learned  that  while  members  of  the  management/evaluation 
staff  were  enrolled  in  graduate  degree  programs,  they  i 

I 

actively  discouraged  the  CBE  authors  from  undertaking  addi- 
tional coursework  themselves.  Even  though  in  some  cases 
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categorized  as  self-serving.  (The  completion  of  additional 
coursework  facilitated  promotion.)  Such  criticism  appeared 
to  the  authors  to  be  hypocritical  in  view  of  the  fact  that 
some  members  of  the  management/evaluation  staff  were 
pursuing  degrees  which  would  enhance  their  professional 
standing.  Despite  these  discouraging  conditions,  after 
appealing  to  GPVC  supervisors,  at  least  two  of  the  original 
eight  authors  took  coursework  while  working  for  the  service 
test . 

The  question  of  whether  or  not  the  authors’  perceptions 
were  accurate  is  not  relevant  to  this  report.  What  is 
important  is  how  they  perceived  the  managers  of  the  service 
test.  That  authors'  perceptions  were  as  described  is  a 
token  of  the  poor  esprit  de  corps  of  the  service  test  per- 
sonnel. To  outside  observers  of  the  project,  the  morale 
usually  appeared  bad.  Such  an  assessment  seems  to  be  corro- 
borated by  the  service  test  manager/evaluators  who  once,  in 
conversation,  characterized  the  attitude  of  the  authors  as 
one  of  "open  rebellion"  to  the  administration  of  the 
project . 

Lesson  Development  Procedures 

Those  interested  in  planning  a CBE  curriculum  project 
would  never  consciously  imitate  the  lesson  development  pro- 
cedures employed  at  Chanute  before  May,  1974.“’  These 

5pia  nners  for  the  PLATO  projects  at  other  military 
training  centers  frequently  included  an  inquiry  into  the 
history  of  the  Chanute  service  test  as  an  essential  part  of 
their  preparations.  That  this  project  was  beset  with  many 
problems  was  well  known.  They  hoped  that  a thorough  know- 
ledge of  these  troubles  would  help  them  to  avoid  similar 
problems  in  their  own  projects. 


procedures  were  the  outcome  of  several  different  influences 
ranging  from  defects  inherent  in  the  establishment  of  the 
project  to  inexperience  in  curriculum  and  instructional 
design.  It  would  be  a difficult  task  and  of  little  use  to 
recount  the  various  attempts  made  at  Chanute  to  bring  some 
order  to  the  production  of  lessons.  The  task  would  be  dif- 
ficult because,  as  with  the  implementation  plans,  several 
different  procedures  were  tentatively  adopted  and  of  little 
value  since  the  administration  of  these  procedures  was 
ineffective. 

The  most  instructive  view  of  these  practices  would  be 
that  which  sees  them  as  a natural  outcome  of  deficiencies  in 
the  establishment  and  management  of  the  project.  This  view 
will  be  adopted  here.  No  attempt,  therefore,  will  be  made 
to  describe  completely  the  development  of  a CEE  lesson  from 
the  planning  stages  to  its  operational  use  or  to  enumerate 
the  number  of  different  lesson  development  procedures 
adopted.  Unless  otherwise  noted,  any  specific  mention  of 
procedures  will  refer  to  those  employed  during  the  tenure  of 
the  last  TRAB  project  officer  who  managed  the  service  test 
for  the  major  portion  of  time  before  the  project  review  in 
May,  1974. 

During  the  phase  of  the  Chanute  service  test  under 
discussion,  there  were  no  timelines  imposed  on  lesson  devel- 
opment by  the  project  management.  Rather,  each  author 
worked  at  his  own  speed  on  a PLATO  lesson  until  it  reached  a 
state  of  perfection  that  was  satisfactory  to  him.  With  no 
timelines  to  be  met,  lesson  development  was  quite  slow.  A 
factor  lengthening  development  time  was  the  fact  that  there 
were  no  precisely  stated  or  enforced  procedures  for  lesson 
development.  Thus,  the  authors  became  the  sole  judges  of 
when  their  lessons  were  ready  for  classroom  use.  The  policy 
of  allowing  the  authors  to  determine  when  their  lessons  are 
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completed  is  particularly  unfortunate  in  the  early  stages  of 
a project.  Novice  authors  frequently  tend  to  be  overly  con- 
cerned with  cosmetic  and  generally  nonessential  aspects  of 
their  lessons  when  they  are  not  constrained  by  timelines  for 
lesson  production.  Without  such  timelines,  an  author’s 
boredom  is  often  the  major  criterion  for  when  a lesson  is 
completed . 

Such  was  the  case  at  Chanute  where  it  was  thought  to  be 
useful  "to  determine  how  long  it  would  take  novice  program- 
mers and  lesson  designers  to  program  one  student  contact 
hour  (Green,  1973,  p.  27).  For  the  project's  first  lessons, 
each  author  was  asked  to  select  a topic  from  a given  portion 
of  the  GPVC  for  a PLATO  lesson  and  then  to  begin  the  task  of 
designing  and  programing  the  lesson.  Working  independently, 
the  authors  took  an  average  of  650  hours  to  complete  one 
student  contact  hour  of  CBE  material  (Green  ^ £]^.,  1973, 
p.  38-j).  Whether  or  not  this  figure  includes  the  time 
needed  for  trial  student  runs  and  lesson  revision  is  not 
clear.  If  it  does  not,  the  actual  time  needed  to  ready  one 
student  contact  hour  by  the  Chanute  PLATO  authors  may  have 
been  as  high  as  700  hours. 

Even  for  a project's  first  lessons,  650  hours  of 
authoring  time  for  one  student  contact  hour  is  an  unusually 
slow  production  rate.  The  Aberdeen  Proving  Grounds  required 
about  400  hours  per  student  contact  hour  for  their  first 
lessons.  Aberdeen's  first  lessons  were  produced  under  simi- 
lar circumstances  to  those  of  Chanute  with  each  author 
being  a CBE  and  instructional  design  novice  and  undertaking 
virtually  all  of  a lesson's  development  (Himwich,  1977). 

Related  to  the  long  lesson  development  times  experience 
during  the  first  phases  of  the  project  was  the  fact  that 
there  were  no  established  lesson  structures.  Instead,  since 
the  project  officer  perceived  the  major  objective  of  the 
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test  to  be  research,  practically  each  lesson  became  an 
experiment  in  instructional  design.  These  experiments  were 
frequently  designed  by  the  individual  author  with  only  the 
approval  of  the  management/  evaluation  staff  which  was  no 
more  experienced  in  the  area  of  instructional  design  than 
the  author.  Thus,  with  one  lesson  designed  primarily  to 
test  the  PLATO  system's  graphics  capabilities,  another  to 
experiment  with  simulation,  another  to  test  the  effective- 
ness of  microfiche,  etc.,  the  CBE  courseware  took  on  a some- 
what random  appearance.  This  randomness  extended  not  only 
to  the  the  basic  intent  of  the  lesson  but  also  the  conven- 
tions employed  by  each  lesson  for  proceeding  through  that 
lesson  and  the  requirements  for  its  successful  completion  by 
the  student.  Thus,  the  first  lessons  taken  as  a whole  did 
not  have  an  appearance  of  belonging  to  the  same  course. 

Although  these  remarks  have  been  made  in  reference  to 
the  project's  first  lessons,  they  apply  in  general  to  all 
the  lessons  produced  during  the  period  of  time  under  discus- 
sion. For  the  most  part,  each  lesson  was  designed  and 
programmed  to  test  some  aspect  of  CBE  that  intrigued  its 
author.  The  PLATO  authors  were  allowed  to  take  the  initia- 
tive in  choosing  the  topics  of  their  lessons  from  the  GPVC 
material,  in  determining  what  capability  of  CBE  was  to  be 
tested  by  their  lesson,  and  in  designing  the  lesson  to  meet 
the  experimental  objectives.  The  role  of  the  management/ 
evaluation  staff  during  this  time  was  primarily  advisory. 
Their  advice  apparently  did  not  include  guidelines  that 
would  have  given  the  lessons  the  appearance  of  belonging  to 
the  same  project  or  timelines  for  how  long  a given  lesson 
should  be  worked  on. 

That  the  authors  worked  in  such  a permissive  atmosphere 
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absence  of  service  test  objectives  more  precise  than  a gen- 
eral charter  to  research  aspects  of  CBE.  The  second  was  the 
lack  of  experience  on  the  part  of  the  management/evaluation 
staff  in  the  crucial  areas  of  curriculum  design,  educational 
research,  and  management.  With  vague  objectives  and  little 
of  the  expertise  needed  to  direct  a research-oriented  cur- 
riculum development  project,  a too-heavy  burden  was  put  on 
the  management/evaluation  staff  of  the  service  test. 

The  authors  had  become  very  used  to  working  indepen- 
dently early  in  the  project.  If  there  were  any  attempts  to 
impose  order  on  the  lesson  development  processes  such 
attempts  would  naturally  be  met  with  resistance.  In  an 
interview,  management/evaluation  staff  expressed  the  view 
that  the  PLATO  authors  expected  some  justification  for  any 
change  in  procedure  they  were  asked  to  make.  For  such 
authority,  the  management/evaluation  staff  had  neither  an 
implementation  plan,  nor  relevant  experience,  nor  precedent 
to  offer.  As  a result,  their  attempts  for  an  active  manage- 
ment of  the  project's  lesson  production  were  for  the  most 
part  ineffectual. 

Summary 

There  is  no  known  record  of  the  output  of  the  Chanute 
PLATO  service  test  during  its  first  21  months.  Several 
reports  indicate  the  progress  at  various  times  before  May, 
197^,  but  no  summative  report  exists.  Projecting  such  data 
from  the  first  annual  report  (Green,  1973),  suggests  that 
the  first  phase  of  the  service  test  produced  about  22  stu- 
dent contact  hours  of  CBE  instruction.^  Allowing  2 months 

^The  estimate  of  the  total  number  of  student  contact 
hours  is  derived  by  projecting  the  ratio  of  the  total  number 
of  manhours  available  with  the  total  number  of  student  con- 
tact hours  produced  as  of  May,  1973. 
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of  training  time  for  the  eight  authors,  the  rate  of  CBE  les- 
son production  was  about  1120  manhours  for  each  student  con- 
tact hour  produced. 7 Also,  80  students  were  used  in  testing 
these  lessons  as  part  of  their  formative  development.  There 
is  no  record  of  any  of  these  lessons  being  used  without  mod- 
ification as  mainline  instructional  materials  at  Chanute  or 
as  part  of  structured  research.  However,  16  of  the  lessons 
were  modified  to  be  consistent  with  the  objectives  of  the 
second  phase  and  were  included  among  the  33  PLATO  modules 
used  in  that  phase. 

This  estimate  of  the  productivity  of  the  first  period 
of  Chanute' s service  test  roughly  agrees  with  those  of  some 
service  test  staff  and  those  of  outside  programming  and  edu- 
cational consultants.  Even  by  doubling  these  output  rates, 
the  productivity  would  have  to  be  considered  poor.  In  dis- 
cussing the  management,  several  of  the  factors  contributing 
to  this  poor  record  have  been  mentioned.  In  summary,  these 
factors  were: 

1)  the  absence  of  precisely  articulated  pro- 
ject objectives, 

2)  the  attempt  to  maintain  a simultaneous 
research  and  operational  orientation  to 
the  project, 

3)  uncertain  designation  of  leadership  fol- 
lowed by  Inexperienced  leadership , 

the  absence  of  structured  CBE  lesson  pro- 
duction procedures, 

5)  slow  hardware  acquisition  and  PLATO  system 
unreliability  during  the  first  six  months 
of  the  service  test. 


"^It  is  interesting  to  note  that  the  report  from  which 
these  numbers  are  derived  gives  445  manhours  as  the  time  to 
develop  one  student  contact  hour.  Thus,  as  a rough  esti- 
mate, the  authors  were  spending  more  than  half  of  their 
time  in  activities  not  directly  related  to  CBE  lesson  pro- 
duction. 
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Although  no  significance  is  intended  in  the  order  of  presen- 
tation of  these  factors,  interviews  with  management  person- 
nel have  indicated  a strong  feeling  that  the  first  was  the  ' 

major  contributor  to  the  service  test's  lack  of  productivity 
during  this  time.  While  this  factor  is  surely  important  in 
this  respect,  it  seems  clear  that  all  five  of  these  factors 
contributed  synergistically  to  the  service  test's  lack  of 
productivity.  Nevertheless,  the  presence  of  any  one  of 
these  would  have  severely  diminished  any  project's  chances 
for  success. 

Final  responsibililty  for  the  absence  of  useful  activ- 
ity during  the  test's  first  phase  must  rest  with  AFHRL  and 
ATC.  Had  they  pursued  more  actively  the  tasks  of  settling 
on  and  supporting  an  implementation  plan  early  in  the  ser- 
vice test  a substantial  portion  of  the  18  month  first  phase 
might  have  been  salvaged.  Such  a course  would  have  contri- 
buted to  a positive  esprit  de  corps  by  showing  high  bureau- 
cratic interest  in  addition  to  giving  the  project  a sense  of 
direction  which  by  itself  might  have  boosted  morale.  Simi- 
larly, ATC  and  AFHRL  were  in  a position  to  insure  that  the 
management/evaluation  staff  was  sufficiently  experienced  to 
I carry  out  the  difficult  task  of  administrating  and  monitor- 

• ing  the  test.  In  May,  1974,  when  ATC  and  AFHRL  finally 

I determined  to  closely  examine  the  test  and  take  a more 

active  hand  in  it,  the  test  was  put  under  experienced 
^ leadership  and  entered  its  most  productive  period. 

f I 


ISD  MAMAGEMEHT  OF  THE  SERVICE  TEST 


Early  in  197^,  the  problems  of  the  Chanute  service  test 
came  to  the  attention  of  the  agencies  which  were  materially 
concerned  with  the  test's  outcome.  A formal  review  at 
Chanute  AFB  was  convened  on  May  15,  197^,  with  reprensenta- 
tives  from  ATC,  AFHRL,  ARPA,  the  Educational  Testing  Ser- 
vice, and  CERL  attending.  The  presentation  offered  by  the 
service  test  personnel  reflected  accurately  the  fact  that 
the  test  had  been  up  to  that  time  a disorganized  effort  and 
that  little  useful  information  about  CBE  had  been  gathered. 
As  a result  the  service  test  was  put  under  new  management 
and  a new  research  agreement  was  drawn  up. 

The  new  research  agreement  called  for  the  Department  of 
Weapons  Systems  Support  Training  (WSST)  Instructional 
Systems  Development  (ISD)  chief  to  take  over  the  management 
of  the  PLATO  authors  from  TRAB.  This  individual  was  already 
revising  the  Special  Purpose  Vehicle  Maintenance  Courses 
(SPVO.®  It  was  felt  that  by  using  the  PLATO  system  to  pre- 
sent the  SPVC  as  revised  by  the  ISD  effort  a measure  of 
the  "true  costs  and  capabilities  of  PLATO  IV  in  a systems 
approach  to  Air  Force  technical  training''  could  be  obtained 
(AFHRL,  197^). 

The  new  research  agreement  called  for  the  ISD  team  to 
integrate  PLATO  lessons  into  the  SPVC.  These  lessons  would 
then  be  offered  with  the  other  revised  components  of  the 
course  as  operational  training  materials.  The  management 
plan  for  the  ISD  effort  states  explicitly  that  "PLATO 
lesson/learning  strategies  and  methodologies  will  be  limited 

^Like  the  GPVC,  the  SPVC  had  several  informal  verbal 
designations.  The  official  name  of  this  sequence  of  courses 
is  3ABR47231-1/A/B/C. 
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to  presently  available  hardware  and  programming  techniques" 
(Gissing,  197^4) . Thus,  rather  than  regard  the  PLATO  system 
as  a research  tool,  in  the  new  phase  of  the  service  test  it 
would  be  used  alongside  other  instructional  media  in  an  on- 
going course. 

The  management  plan  called  for  the  development  of  mate- 
rials to  support  88  training  objectives.  Of  these,  50 
objectives  were  knowledge  oriented,  and  33  of  the  knowledge 
oriented  objectives  were  judged  to  be  suitable  for  PLATO 
support.  Happily,  the  SPVC  and  GPVC  were  not  totally  dis- 
similar in  their  content.  With  some  modifications,  lessons 
from  the  first  phase  of  the  service  test  could  be  used  to 
support  17  of  the  33  objectives.  CBE  lessons  for  the 
remaining  16  objectives  were  completed  in  the  time  allotted 
by  the  management  plan.  Along  with  the  modified  lessons 
from  the  first  phase  of  the  service  test  these  PLATO  lessons 
became  part  of  the  SPVC  in  January,  1975. 

Comparison  of  the  First  and  Second  Phases 

Because  it  was  the  only  phase  of  the  service  test  that 
produced  a substantial  amount  of  analyzable  data  on  CBE  use 
in  military  training,  the  second  phase  has  been  extensively 
documented  in  reports  by  the  Air  Force  and  CERL.  To  report 
on  the  same  aspects  of  the  second  phase  in  the  detail  that 
was  lavished  on  the  first  would  largely  duplicate  what  has 
already  been  written.  For  this  reason,  a bibliography  of 
these  reports  with  a brief  description  of  their  contents  is 
included  in  this  section.  To  supplement  these  reports  and 
to  continue  with  the  discussion  of  the  entire  service  test, 
a comparison  of  important  aspects  of  the  first  two  phases 
will  be  drawn. 

An  important  difference  between  the  first  and  second 
phases  of  the  Chanute  service  test  was  that  the  second 
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phase's  goals  and  means  of  attaining  them  were  carefully  ’ 

articulated  at  the  outset  in  an  implementation  plan.  This  ^ 

i 

plan  laid  down  the  responsibilities  of  all  of  the  parties  , 

involved  with  the  project.  In  particular,  the  responsibil- 
ity for  managing  the  eight  PLATO  authors  was  given  to  the 
ISD  chief  while  the  role  of  TRAB  was  relegated  to  that  of  "j 

liaison  for  the  service  test  with  ARPA,  ATC , AFHRL,  and  J 

CERL.  TRAB  was  also  given  the  responsibility  for  conducting  ] 

tours  of  the  PLATO  project  so  that  there  would  be  no  inter- 
ference with  the  PLATO  authors'  work.  For  the  evaluation 
activities  that  were  the  joint  responsibility  of  the  AFHRL  j 

representative  and  TRAB,  the  plan  specified  that  guidelines  ] 

were  to  be  drawn  up  which  would  preclude  the  evaluation  ^ 

procedures  from  interfering  with  the  lesson  development. 

By  simply  possessing  a management  plan,  the  ISD  phase 
of  the  service  test  differed  substantially  from  the  first 
phase.  In  delineating  responsibilities  of  the  various 
agencies  involved  in  the  test,  the  distinction  from  the 
first  phase  was  still  more  striking.  Added  to  these  differ- 
ences was  the  fact  that  the  management  plan  set  out  time- 
lines for  PLATO  lesson  production  and  validation.  Thus, 
the  new  manager  of  the  service  test  prssessed  some  standard 
by  which  to  govern  the  rate  of  lesson  production.  In  fact,  : 

knowing  the  amount  of  lesson  material  needed  and  the  amount 
already  existing,  he  initially  established  production  rates 
of  50  hours  per  student  contact  hour.  Being  based  on  the  / 

production  rates  for  programmed  texts,  this  expectation  was, 
perhaps,  overly  optimistic.  Nevertheless,  this  goal  did 
establish  an  objective  that  could  be  articulated  to  the 
authors  and  could  be  used  as  a touchstone  to  determine  pro- 
gress. Without  such  a management  device,  attempts  at  ] 

administering  a curriculum  development  effort  are  likely  to  ' 

% 

be  ineffectual,  as  was  demonstrated  in  the  first  phase.  ] 
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During  the  first  phase,  the  administration  of  the  ser- 
vice test  was  placed  in  hands  inexperienced  in  both  curricu- 
lum development  and  sound  management  practices.  The  new 
manager,  as  ISD  chief,  was  responsible  for  revising  107 
existing  courses  to  improve  instructional  efficiency  while 
at  least  maintaining  student  performance.  When  he  undertook 
to  manage  the  PLATO  service  test,  he  had  already  carried  out 
revision  of  about  32  of  these  courses.  In  a total  of  22 
years  at  the  School  of  Applied  Aerospace  Sciences,  he  had 
occupied  a variety  of  positions  from  instructor  to  WSST  ISD 
team  chief.  He  had  spent  seven  years  with  the  ISD  team  and 
two  years  as  its  chief.  Thus,  he  possessed  extensive 
experience  in  the  designing  and  administration  of  curriculum 
development  projects. 

Another  striking  difference  between  the  first  two 
phases  of  the  service  test  was  the  manner  in  which  curricu- 
lum and  lesson  design  was  carried  out.  During  the  first 
phase,  each  PLATO  author  carried  out  for  himself  most  of  the 
activities  needed  in  CBE  lesson  production.  These  activi- 
ties included  lesson  design,  TUTOR  programming,  providing 
subject  matter  knowledge,  and  lesson  validation.  These 
requirements,  difficult  to  meet  by  even  the  most  experienced 
educators,  were  beyond  the  qualifications  of  the  PLATO 
authors.  As  part  of  an  ISD  project,  however,  the  authors 
joined  a more  integrated  and  disciplined  effort  with  defined 
roles  and  procedures.  While  they,  in  fact,  designed  most  of 
the  CBE  lessons,  their  primary  responsibility,  in  the  view 
of  the  ISD  team  chief,  was  to  provide  TUTOR  programming 
expertise.  The  details  of  the  ISD  curriculum  design  model 
are  given  in  several  of  the  reports  summarized  after  the 
discussion  of  the  ISD  phase. 

Under  ISD  management,  the  service  test’s  orientation 
was  very  different  from  what  it  had  been  under  TRAB 
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management.  Whereas  before  the  test  was  attempting  to 
research  basic  CBE  capabilities,  the  second  phase  was  simply 
motivated  to  develop  training  materials  that  would  be  used 
as  part  of  operational  training.  Being  operationally 
oriented  did  not  permit  the  luxury  of  selecting  the  most 
interesting  topics  to  teach  via  the  PLATO  system  or  of 
determining  experimentally  how  those  topics  might  be  most 
effectively  taught.  In  practical  terms,  the  shift  to  the 
production  of  CBE  materials  determined  that  the  PLATO 
authors  could  no  longer  play  a major  role  in  directing  their 
own  work. 

Under  the  new  management,  CBE  lesson  production 
improved  markedly.  While  no  good  records  were  kept,  the  ISO 
chief  estimates  lesson  production  at  100  manhours  per  stu- 
dent contact  hour.  Keeping  in  mind  that  the  PLATO  authors 
produced  as  much  CBE  lesson  material  in  six  months  under  the 
ISD  management  as  they  had  produced  in  over  18  months  under 
TRAB  management,  it  is  clear  that  this  estimate  is  at  least 
plausible.  The  reader  must  be  warned,  however,  that  these 
estimates  are  valid  only  if  taken  as  general  comparisons  in 
productivity  between  the  two  phases;  no  careful  documenta- 
tion of  lesson  production  rates  for  either  phase  is 
available . 

Finally,  the  ISD  effort  has  left  at  Chanute  AFB  a tan- 
gible product.  The  CBE  lessons  altered  from  the  first 
phase's  work  or  developed  during  the  second  phase  are  still 
being  used  in  operational  training  at  Chanute.  Since  Janu- 
ary, 1975,  the  CBE  lessons  required  for  the  SPVC  have  been 
used  by  an  average  of  350  students  per  year  for  approxi- 
mately 7000  student  contact  hours.  By  itself,  this  outcome 
is  praiseworthy  for  its  contribution  to  military  training  as 
well  as  for  the  accumulation  of  data  giving  evaluators  some 
insight  to  the  potentials  and  problems  of  CBE  use  in 
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military  technical  training.  In  comparison  to  the  first 
phase,  this  accomplishment  is  all  the  more  impressive  for 
its  magnitude  and  the  efficiency  of  its  execution. 

Reports  Describing  the  ISP  Phase  of  the  Service  Test 

The  following  reports  and  articles  discuss  various 
aspects  of  the  Chanute  service  test  from  May,  1974,  to 
April,  1975.  Here,  the  title,  author  and  a brief  descrip- 
tion of  the  contents  of  the  document  are  described.  Full 
bibliographical  information  for  each  of  these  publications 
may  be  found  in  the  List  of  References. 

An  Overview  of  Chanute  Lessons  by  J.  A.  Klecka  (1977a). 
An  analysis  and  summary  of  eight  PLATO  lessons  written 
during  the  ISO  phase  of  the  service  test.  The  study  is 
based  on  in-depth  reviews  of  these  lessons  both  in  the  form- 
ative and  final  stages  of  development.  An  indication  of 
general  characteristics  of  ISO  PLATO  lesssons  is  given.  The 
study  concludes  with  an  assessment  of  the  instructional 
effectiveness  and  utilization  of  the  instructional  capabili- 
ties of  these  lessons. 

Three  Aspects  of  PLATO  Use  at  Chanute  AFB  by  J.  A. 
Klecka  (1977b).  The  first  of  these  chapters  describes  the 
implementation  of  an  ISO  systems-design , team-authored  PLATO 
lesson  production  process;  the  second,  the  use  of  the  PLATO 
system  for  data  management,  student  testing,  and  other 
related  computer  managed  instructional  uses;  the  last  exam- 
ines the  changes  made  to  the  eight  PLATO  lessons  sampled  in 
Klecka  (1977a)  in  order  to  achieve  the  validation  criterion. 

Evaluation  of  PLATO  IV  in  Vehicle  Maintenance  Training 
by  B.  E.  Dallman,  P.  J.  Deleo,  P.  S.  Main,  G.  C.  Gillman 
(1977).  While  nominally  covering  the  entire  Service  Test, 
this  report  of  about  300  pages  deals  mostly  with  the  ISO 
phase.  The  data  accumulated  during  this  phase  is  presented 


and  analyzed.  Additionally,  extensive  anecdotal  information 
concerning  this  phase  is  distributed  throughout  the  report. 
To  obtain  a balanced  view  of  much  of  the  subject  matter  of 
this  report,  its  reading  should  be  accompanied  by  the  read- 
ing of  Misselt  ^ (1977). 

"A  Critique  of  the  Final  Evaluation  Report  of  the 
Computer-based  Education  Service  Test  at  Chanute  AFB"  by 
A.  L.  Misselt,  H.  A.  Himwich,  L.  D.  Francis,  R.  A.  Avner, 

K.  K.  Tatsuoka,  and  J.  A.  Klecka  (1977).  The  critique  is  a 
close  chapter-by-chapter  commentary  on  Dallman  ^ al . 

(1977).  This  commentary  was  written  by  CERL  personnel  who 
acted  as  evaluation,  instructional,  and  programming  consul- 
tants for  the  Chanute  CBE  service  test. 

"Military  Instructor  Attitudes  Toward  Computer-Based 
Technical  Training”  by  J.  A.  Klecka  (1977c).  The  attitudes 
of  Chanute  instructors  toward  the  PLATO  system  in  vehicle 
maintenance  training  were  investigated.  It  was  believed 
that  instructor  atttitudes  would  influence  instructor  and 
student  acceptance  of  CBE  in  the  training  environment.  The 
attitudes  were  measured  via  surveys  and  interviews  with  the 
instructor  staff  and  other  related  personnel  at  Chanute  over 
a period  of  several  months.  The  results  gave  a basically 
positive  picture  of  instructor  attitudes  and  their  conse- 
quent impact  on  student  interaction  with  the  PLATO  IV 
system . 

"Attitudes  and  Performance  of  Military  Students  in 
Computer-Based  Technical  Training"  by  K.  K.  Tatsuoka,  A.  L. 
Misselt,  and  P.  L.  Maritz  (1977).  The  student  attitude  data 
reported  by  Dallman  ^ al^.(1977)  are  re-analyzed  and  com- 
pared with  measures  of  performance.  The  results  and  inter- 
pretations differ  somewhat  from  those  reported  by  Dallman 
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Summary 

The  ISD  phase  of  the  Chanute  service  test  was  in  many 
respects  a new  beginning  for  the  test.  Although  the  staff 
of  PLATO  authors  remained  the  same,  a new  research  agreement 
initiated  the  phase  along  with  new  management,  new  objec- 
tives and  a new  approach  to  CBE  lesson  development.  The 
result  was  an  eight-month  effort  which  was  more  productive 
in  terms  of  CBE  lesson  material  written  and  validated,  CBE 
materials  implemented  in  technical  training,  and  information 
concerning  CBE  use  in  military  training  than  the  rest  of  the 
four  year  service  test  combined.  That  Dallman  et  al . ( 1 977 ) 
frequently  treat  the  ISD  phase  of  the  service  test  as  the 
service  test  and  devote  substantially  more  than  half  of  the 
total  pages  of  the  final  report  to  it  may  be  taken  as  a 
token  of  the  relative  importance  of  the  ISD  phase. 

As  Klecka  (197Ta)  points  out,  the  CBE  lessons  developed 
during  the  ISD  phase  were  not  exemplary  in  terms  of  using 
the  PLATO  medium  in  an  effective  manner.  This  fact  may  be 
in  part  accounted  for  by  the  fact  that,  despite  extensive 
experience  in  curriculum  development  in  more  conventional 
media,  the  ISD  team  had  little  experience  with  CBE.  More- 
over, the  time  constraints  that  the  ISD  team  were  under  dic- 
tated that  elaborate,  media-sensitive  PLATO  lessons  were  an 
unessential  luxury.  Finally,  in  the  development  of  any 
instructional  materials,  the  ISD  team  endeavored  to  design 
the  materials  to  teach  only  what  was  required  and  no  more. 
The  fact  that  the  ISD  PLATO  lessons  did  not  represent  opti- 
mal uses  of  CBE  does  not  greatly  detract  from  the  achieve- 
ment of  actually  producing  a large  body  of  instructional 
material.  This  accomplishment,  substantial  in  itself,  is 
the  more  impressive  within  the  context  of  the  rest  of  the 
service  test. 


Although  it  would  be  perhaps  impossible  to  attribute 
the  relative  success  of  the  ISD  phase  to  any  one  particular 
cause,  it  is  plausible  that  one  or  more  of  the  differences 
between  the  TRAB  and  ISD  phases  of  the  project  was  the 
source.  The  major  differences  are  summarized  below. 

1)  An  implementation  plan  governed  the  ISD 
phase.  The  plan,  which  had  the  approval 
of  ATC  and  AFHRL,  precisely  articulated 
objectives  and  means  and  timelines  for 
achieving  them.  The  TRAB  phase,  on  account 
of  an  inability  to  obtain  agreement  from 
ATC  and  AFHRL,  possessed  no  finally- 
approved  Implementation  plan. 

2)  The  PLATO  authors  were  directed  by  an 
experienced  curriculum  developer  during 
the  ISD  phase;  during  the  TRAB  phase,  the 
managers  were  novices  in  curriculum  devel- 
opment, CBE , and  management.  Again  this 
would  seem  to  be  due  to  casual  interest  in 
the  service  test  from  AFHRL  and  ATC. 

3)  During  the  ISD  phase,  CBE  materials  were 
developed  in  accordance  with  validated 
instructional  development  procedures. 

In  the  ISD  development  procedures  the 
PLATO  authors  filled  an  important  but 
precisely  circumscribed  role.  Under  TRAB 
management,  the  PLATO  authors  were  respon- 
sible for  virtually  every  facet  of  CBE 
lesson  development.  Moreover,  lesson 
development  procedures  were  left  to  the 
discretion  of  individual  authors. 

That  these  differences  existed  between  the  two  phases 

is  a matter  of  fact.  That  one  or  more  of  them  could  explain 
the  striking  difference  in  outcomes  between  the  two  phases 
appears  to  be  a matter  of  common  sense.  In  addition  to 
common  sense,  interviews  with  service  test  personnel  bolster 
the  conclusion  that  these  factors  were  important  in  deter- 
mining the  relative  success  or  failure  of  each  phase. 
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Foreword 

The  following  comments  partially  fulfill  a contractual 
obligation  between  the  Defense  Advanced  Research  Projects 
Agency  (ARPA)  and  the  Computer-based  Education  Research  Lab- 
oratory of  the  University  of  Illinois  (CERL)  to  provide  a 
critique  of  reports  produced  at  the  various  PLATO  sites  sup- 
ported by  ARPA.  The  comments  supplement  the  communications 
exchanged  between  Chanute  AFB  c^d  CERL  during  the  course  of 
a long  and  close  working  relationship.  They  are  intended  to 
give  the  reader  a view  of  the  project  in  the  context  of  the 
general  problem  of  implementation  of  new  instructional  tech- 
nology as  well  as  providing  independent  comments  on  histori- 
cal and  procedural  matters.  The  comments  are  based  on  the 
experience  of  the  CERL  staff  in  aiding  the  implementation 
and  operation  of  over  20  military  Computer  Based  Education 
(CBE)  sites,  as  well  as  professional  expertise  in  the  areas 
of  military  instruction,  instructional  design,  and  evalua- 
tion . 


We  were  able  to  prepare  this  critique  because  the  Air 
Force  Human  Resources  Laboratory  (AFHRL)  staff  kindly  made 
their  report  available  to  us  in  draft  form.  Unfortunately, 
contract  completion  dates  are  such  that  publication  of  this 
critique  must  be  made  before  the  final  version  of  the  AFHRL 


report  (Dallman,  DeLeo,  Main,  & Gillman,  1977)  will  be 
available.  While  knowing  that  the  comments  we  and  other 
reviewers  have  made  may  affect  the  final  contents  of  the 
AFHRL  report,  time  requirements  have  forced  us  to  base  our 
remarks  on  the  draft  version  available  in  April,  1977. 


Therefore,  readers  may  expect 


our  critique 
final  version 


to  find  that  some  problems 
raises  have  been  corrected 


The  organization  of  this  critique  parallels  that  of  the 
report;  comments  are  offered  section-by-section.  In  writing 
the  critique,  we  have  made  the  assumption  that  the  reader  has 
a copy  of  the  AFHRL  report  in  hand  in  order  to  follow  the 
commentary;  we  have  provided  only  a brief  restatement  of  the 
procedures  and  findings  on  each  topic.  To  facilitate  refer- 
ences to  the  AFHRL  report,  the  term  "chapter"  is  used  to 
denote  the  major  sections  identified  by  Roman  numerals. 

We  have  found  the  AFHRL  report  to  be  well-organized  and 
informative.  It  is  an  important  contribution  to  the  liter- 
ature on  CBE  and  on  the  implementation  of  curricular  innova- 
tions. All  of  those  who  assisted  in  the  implementation  and 
reporting  of  the  Chanute  CBE  project  have  shown  commendable 
effort  and  professional  expertise  in  meeting  the  problems  of 
testing  new  technology  in  a realistic  training  environment. 


I.  INTRODUCTION 


Project  Genesis 

The  section  titled  "Project  Genesis"  is  a brief  but 
accurate  summary  of  the  origins  of  the  service  test  at 
Chanute.  It  should  be  noted,  however,  that  in  addition  to 
stating  the  broad  objectives  for  the  service  test,  the  ARPA/ 
AFHRL/ATC  research  agreement  (cited  in  this  section)  also 
outlined  an  implementation  plan  for  achieving  those  objec- 
tives. It  called  for  the  development  of  computer-based 
instructional  materials  on  the  PLATO  system  which  would 
duplicate  the  content  of  Chanute's  General  Purpose  Vehicle 
Course  (GPVC).  With  identical  content  for  that  course  avail 
able  in  both  computer-based  and  conventionally-delivered 
modes,  it  would  have  been  possible  to  conduct  a comparison 
of  the  two  media  categories.  Under  that  plan  the  conven- 
tional course  would  have  provided  a standard  against  which 
the  "cost  effectiveness,  acceptance,  incor porability  (into 
ongoing  technical  training),  and  reliability"  could  be  inves 
tigated.  For  reasons  described  later  in  the  report  and  in 
this  critique,  that  initial  plan  proved  to  be  infeasible. 

Responsibilities 

The  delineation  of  responsibilities  for  AFHRL  and  ARPA 
provides  a concise  description  of  their  roles.  However,  the 
role  of  the  Training  Research  Applications  Branch  (TRAB)  is 
characterized  in  the  original  Research  Training  Agreement  as 
the 

"Technical  School  OPR  [Office  of  Primary 
Responsibility]  for  coordination  of  the 
experimental  service  test  with  ATC/XPT,  AFHRL, 

ARPA,  and  the  University  of  Illinois.  This 
office  will  monitor  the  research  program  by 
developing  a programmed  action  directive  for  the 
duration  of  the  three-year  research  agreement." 
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While  the  full  intent  of  this  statement  is  not  altogether 
clear,  it  seems  to  imply  a somewhat  larger  role  than  the 
responsibility  ascribed  to  the  TRAB  in  Chapter  I,  i.e.,  "to 
evaluate  and  document  the  project  outcomes". 

The  difficulty  in  describing  the  TRAB's  role  is  that 
the  Chanute  service  test  actually  consisted  of  three  distinct 
"projects",  which  are  referred  to  throughout  the  report  as 
"phases".  Since  each  phase  had  objectives  and  management 
unlike  that  of  the  others,  it  is  possible  to  think  of  them  as 
different  projects.  The  responsibilities  outlined  in  this 
section  of  Chapter  I are  true  of  the  second  project  or  Phase 
II;  they  do  not  necessarily  describe  the  roles  taken  by  the 
various  agencies  and  units  (especially  that  of  the  TRAB) 
during  the  other  phases. 

Also,  while  it  is  clear  that  it  was  ATC's  responsibility 
to  provide  manpower  and  facilities,  it  is  apparent  from  the 
discussion  in  Chapter  II  under  the  heading  "Selecting  an 
Evaluation  Paradigm"  (as  well  as  the  experience  of  CERL)  that 
ATC  had  a larger  policy-making  role. 

Project  Personnel 

Except  that  it  fails  to  mention  the  participation  of 
members  of  the  instructional  systems  development  (ISD)  team 
during  Phase  II,  the  description  of  the  service  test  staffing 
is  correct.  All  of  the  personnel  described  in  this  section 
did  contribute  to  the  project,  but  they  were  not  all  present 
simultaneously  or  throughout  its  duration. 

Scope  of  Development 

The  development  described  in  this  section  was  that  of 
Phase  II  of  the  service  test.  Phases  I and  III  are  described 
separately  in  Chapter  II. 
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II.  BACKGROUND 
PLATO  System  Description 

The  description  of  the  PLATO  system  is  brief,  but  accu- 
rate. Readers  wishing  a more  complete  background  may  refer 
to  the  article  by  Smith  and  Sherwood  (1976)  or  the  bibliogra- 
phy by  Lyman  ( 1 977  ) . 

Historical  Perspective 

Phase  I.  The  discussion  of  the  events  of  Phase  I is  a 
highly  abbreviated  description  of  the  activities  and  deci- 
sions of  this  period.  Some  additional  information  is 
provided  in  the  following  comments. 

The  first  implementation  plan  for  the  service  test  (con- 
tained in  the  ARPA/AFHRL/ATC  research  agreement)  called  for 
the  comparison  of  CBE  instructional  materials  with  conven- 
tional materials  already  in  use  in  the  GPVC.  Hence  the 
personnel  expected  to  develop  the  CBE  materials,  i.e.,  the 
authors,  were  selected  from  among  the  GPVC  instructors.  As 
indicated  in  the  AFHRL  report,  the  eight  individuals  who  were 
selected  varied  widely  in  the  extent  of  their  teaching  exper- 
ience and  knowledge  of  the  subject  matter  to  be  taught,  and 
they  were  uniformly  inexperienced  with  regard  to  development 
of  CBE  materials.  They  received  their  initial  training  in 
the  use  of  the  TUTOR  language  at  CERL  in  the  Fall  of  1972. 

The  TUTOR  workshop  they  attended  was  the  first  of  several 
conducted  by  the  Military  Training  Centers  (MTC)  group  of 
CERL  for  the  ARPA-suppor ted  PLATO  projects.  For  a descrip- 
tion and  evaluation  of  the  MTC  TUTOR  course,  see  the  report 
by  Francis  (1976b). 

As  indicated  by  the  report  authors,  the  lesson  develop- 
ment efforts  during  Phase  I were  exploratory  and  served  to 
allow  the  Chanute  authors  to  expand  their  skills  and  develop 
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their  interests.  Compared  to  Phase  II  lessons,  those  written 
in  Phase  I exploited  system  capabilities  much  further  in 
terms  of  their  wider  use  of  computer-generated  questions, 
individualization  via  feedback  and  branching,  and  animated 
graphics.  Each  of  the  Phase  I lessons  was  considered  by  the 
staff  to  be  a mini-experiment  in  CBE;  each  was  thought  to 
investigate  the  use  of  one  or  more  of  the  features  described 
above.  Developed  by  persons  with  little  or  no  experience  in 
instructional  design  or  CBE,  these  lessons  were  written  with 
idiosyncratic  instructional  strategies  and  often  failed  to 
represent  the  educational  concept  or  CBE  feature  they  were 
designed  to  investigate.  If  the  reader  keeps  in  mind  the 
statement  in  the  report  that  "Each  author  became  an  investi- 
gator in  his  own  right",  and  the  fact  that  these  authors  were 
highly  inexperienced,  one  can  easily  construct  an  image  of 
the  resulting  lessons.  Nevertheless,  experience  was  gained 
in  the  areas  of  lesson  design  and  programming  and  a number 
of  lessons  were  developed,  many  of  which  were  later  modified 
for  use  in  operational  training  during  Phase  II. 

There  were  a number  of  problems  and  frustrations  during 
Phase  I which  hampered  the  productivity  of  the  project.  In 
addition  to  the  constraints  mentioned  by  the  report  authors 
(e.g.,  delays  in  equipment  delivery,  computer  memory  short- 
ages, system  unreliability,  changes  in  and  lack  of  documen- 
tation of  the  TUTOR  language)  , there  were  several  management 
and  environmental  problems  which  interfered  with  the  goals 
of  the  service  test.  These  included  the  lack  of  commitment 
on  the  part  of  certain  operationally-oriented  perr,onnel  to 
the  test’s  exploratory  nature,  turnover  and  inexperience 
among  management  personnel,  and  the  lack  of  an  approved 
implementation  plan.  These  latter  problems  are  discussed  in 
detail  in  the  chapter  by  Himwich  (1977a). 
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By  early  197^  it  was  apparent  that  the  CBE  lessons 
developed  to  replace  selected  classroom  lectures  in  the  GPVC 
did  not  represent  the  most  effective  application  of  the  PLATO 
medium.  This,  plus  the  fact  that  student  flow  through  the 
GPVC  was  to  drop  drastically,  led  to  a suggestion  that  CBE 
be  implemented  in  the  common  area  of  four  Special  Purpose 
Vehicle  Courses  (SPVC).  Not  only  was  it  hoped  that  the 
existing  GPVC  lessons  could  be  utilized  in  the  SPVC,  it  would 
also  be  possible  to  take  advantage  of  the  larger  student  flow 
for  a full  program  of  lesson  validation  and  program  evalua- 
tion. Finally,  since  the  SPV  courses  were  about  to  undergo 
a systematic  revision,  it  would  be  a simple  matter  to  inte- 
grate PLATO  as  a primary  instructional  medium,  hence  simpli- 
fying many  implementation  and  evaluation  problems. 

The  suggestion  to  shift  CBE  development  to  the  SPVC 
was  put  in  the  form  of  a proposal  and  submitted  to  ATC  and 
AFHRL  for  approval.  This  approval  was  granted  following  a 
formal  project  review  in  May  197^.  However,  more  than  just 
a shift  in  target  courses  was  decided.  There  was  also  a 
shift  in  the  general  orientation  of  the  project  from  an 
exploratory  research  focus  to  one  of  integrating  PLATO  into 
operational  training.  This  was  to  be  accomplished  by  assign- 
ing the  authoring  staff  to  the  supervision  of  the  3340th 
Technical  Training  Group's  ISB  team.  The  May  1974  project 
review  marked  the  transition  from  Phase  I to  Phase  II  and 
hence  was  an  important  turning  point  in  the  test's  history. 

Additional  data  on  Phase  I,  the  first  22  months  of  the 
service  test,  would  have  been  useful.  Some  data  are  pre- 
sented in  the  chapter  by  Himwich  (1977a).  Also,  several  pro- 
ject reports  exist  which  supply  some  of  these  data.  The 
most  useful  of  these  is  the  report  by  Green  (1973).  It  gives 
a detailed  description  of  the  authors  selected,  an  extended 
discussion  of  early  management  problems  of  the  service  test 
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of  the  activities  and  outcomes  of  the  service  test's  second 
phase.  The  brevity  is  justified  because  most  of  the  discus- 
sion in  the  remainder  of  the  AFHRL  report  is  devoted  to  data 
collected  during  Phase  II. 

Phase  III.  A third  phase  of  the  test  was  originally 
conceived  as  a way  to  utilize  surplus  authoring  time  avail- 
able during  the  validation  period  for  the  Phase  II  lessons. 
The  contract  period  with  ARPA  ended  on  September  30,  1975, 
but  was  extended  until  June  30,  1976,  to  allow  time  for  fur- 
ther analysis  of  Phase  II  evaluation  data  and  the  completion 
of  the  Phase  III  studies.  The  revised  management  plan  under 
which  Phase  III  was  begun  was  thought  by  CBE  experts  from 
CERL  to  be  overly  ambitious  though  feasible  in  its  main 
intent.  The  management  of  this  portion  of  the  project  was 
as  unstructured  as  that  of  Phase  I,  but  the  exodus  of  trained 
PLATO  authors  was  itself  sufficient  to  make  the  objectives 
unattainable.  A detailed  summary  of  the  staffing  problems 
of  the  service  test  may  be  found  in  the  chapter  by  Himwich 


( 1977a)  . 

Although  never  officially  described  as  such,  a fourth 
service  test  phase  was  funded  by  ATC.  Running  from 
July  1,  1976,  to  September  30,  1976,  it  was  intended  to 
allow  additional  time  to  collect  data  on  the  cost  saving 
contributions  of  the  PLATO  system  to  the  operation  of  the 
SPVC.  Specifically,  the  goals  of  this  follow-on  effort  were 
to  determine  the  time  savings  due  directly  to  PLATO  (i.e., 
above  and  beyond  those  due  to  group-pacing) , to  determine 
the  benefit  from  on-line  testing,  and  to  determine  the  max- 


71 


imum  student  load  that  could  be  handled  with  Chanute's  com- 
plement of  30  PLATO  IV  terminals.  To  answer  this  last  ques- 
tion, the  SPVC  lessons  were  used  as  replacements  for  a number 
of  the  lectures  in  the  GPVC  in  order  to  increase  the  total 
student  flow  through  the  materials.  The  results  of  these 
"Phase  IV"  studies  are  reported  in  an  unpublished  manuscript 
prepared  by  the  former  chief  of  Chanute's  Training  Research 
Applications  Branch.  These  results  are  not  discussed  in  the 
AFHRL  report. 

Instructional  Setting 

The  "instructional  setting"  section  of  the  report  deals 
with  the  relationship  among  the  four  target  courses,  the 
manner  in  which  the  courses  were  changed  as  a result  of  the 
ISD  team's  redesign  of  the  curriculum,  the  basic  student 
performance  indices  that  were  measured,  and  the  general 
nature  of  the  material  selected  for  CBE  development.  This 
information  is  fundamental  to  an  understanding  of  the  Phase 
II  effort  and,  considering  the  small  amount  of  space 
allotted,  it  is  presented  with  admirable  clarity.  To  facil- 
itate interpretation  of  later  sections,  a few  of  the  more 
important  points  deserve  special  attention.  They  are  high- 
lighted in  the  following  comments. 

One  of  the  most  basic  changes  resulting  from  the  ISD 
redesign  of  the  courses  is  that  classes  progressed  through 
the  revised  versions  at  the  rate  of  their  slowest  members 
(i.e.,  group-pacing  was  used).  If  a class  were  heterogeneous 
in  previous  background  and  learning  rate  (the  usual  case 
since  homogeneous  grouping  was  not  attempted),  the  faster 
students  were  assigned  supplemental  material  to  keep  them 
occupied.  Thus,  the  change  to  group-pacing  allowed  more 
flexible  scheduling  than  was  possible  in  the  conventional 
courses,  but  it  did  not  provide  optimum  pacing  for  all  stu- 
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dents.  An  individually  self-paced  system  would  have  been 
ideal,  but  according  to  interviews  with  Chanute  AFB  person- 
nel, such  an  approach  could  not  be  implemented  because  of  a 
shortage  of  laboratory  equipment  and  because  the  bulk  of  the 
instruction  in  the  specialty  shreds  was  presented  via  lec- 
tures. To  develop  individualized  instruction  to  replace  the 
lectures  was  thought  to  have  required  more  effort  than  could 
be  justified  by  the  relatively  snail  student  flow  through  the 
individual  shreds  and  the  expense  of  the  additional  laboratory 
equipment  that  would  be  needed. 

Another  change  affecting  the  instructional  efficiency  of 
the  revised  common  course  segment  relative  to  that  of  the 
conventional  version  was  the  introduction  of  the  "Block  X” 
concept.  The  effective  length  of  the  courses  was  shortened 
by  assigning  a number  of  topics  to  be  studied  individually  as 
homework.  It  is  our  understanding  that  much  of  this  material 
had  been  presented  during  class  time  for  the  "conventional” 
common  course  segment.  In  comparing  the  two  versions, 
therefore,  it  should  be  noted  that  the  revised  segment  had 
less  content  than  did  the  conventional  version. 

The  reversal  in  proportions  of  time  spent  on  "knowledge" 
and  laboratory  tasks  was  due  both  to  decreased  emphasis  on 
the  knowledge  areas  and  an  increased  emphasis  on  the  perform- 
ance skills  taught  in  the  courses.  These  shifts  in  emphasis 
came  as  a result  of  the  job  analysis  conducted  by  the  ISD 
team  as  the  initial  step  in  development.  A second  factor  was 
that  the  lock-step  pacing  in  the  conventional  version  had 
allowed  more  "slack"  in  the  scheduling  of  the  knowledge  areas 
than  in  the  laboratory  activities.  Hence,  there  was  more 
time  to  be  saved  through  group-pacing  in  the  cognitive  por- 
tions . 

Another  item  that  should  be  noted  is  that  the  content  of 


the  block  exams  for  the  revised  common  course  segment  dif- 
fered from  those  used  in  the  conventional  version.  Because 
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of  shifting  emphases  in  the  content  of  the  first  four  blocks 
of  the  courses,  the  block  exams  used  in  the  conventional  ver- 
sion were  not  valid  for  use  in  the  revised  segment.  It  is 
not  possible,  therefore,  to  compare  directly  the  block  exam 
performance  between  students  taking  the  conventional  and 
revised  versions.  Although  the  tests  generally  measure 
similar  topics,  their  coverage  is  sufficiently  different  to 
keep  them  from  being  parallel. 

This  section  of  the  AFHRL  report  gives  only  a brief 
description  of  the  CBE  lessons  produced  during  Phase  II.  A 
more  thorough  presentation  is  available  in  the  report  by 
Klecka  (1977a)  which  gives  a detailed  discussion  of  eight 
Phase  II  lessons  selected  by  an  AFHRL  evaluator.  Each  les- 
son was  reviewed  in  depth  and  the  resulting  comments  were 
synthesized  to  give  an  overall  description  and  assessment 
of  the  lessons  reviewed.  The  Klecka  report  was  used  as  a 
resource  and  database  for  Chapter  VI  of  the  AFHRL  report. 

Selecting  an  Evaluation  Paradigm 

It  is  commendable  that  those  who  planned  the  AFHRL  eval- 
uation systematically  attempted  to  collect  data  on  the  infor- 
mation needs  and  priorities  of  the  several  agencies  connected 
with  the  Chanute  service  test  (i.e.,  the  various  audiences 
for  the  evaluation  report) . The  result  of  this  effort  was 
the  assigning  of  highest  priority  to  those  areas  of  greatest 
interest  to  ATC , the  agency  which  would  have  to  make  deci- 
sions regarding  support  for  future  applications  of  CBE  in 
Air  Force  training. 

In  view  of  ATC ' s requirements  for  data  on  which  to  base 
decisions,  the  evaluators  chose  to  conduct  a comparative 
evaluation  rather  than  a more  generalizable  ”evaluation-as- 
research"  approach  as  might  be  suggested  by  Cronbach  (1963). 

A comparative  evaluation  usually  seeks  to  provide  data  for 
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selection  of  one  of  the  specific  products  being  evaluated, 
but  confounding  of  subject-matter,  student  population,  and 
instructional  design  variables  with  the  media  used  for 
instruction  would  limit  the  generalizability  of  a single 
comparative  study.  Recognizing  these  problems,  the  authors 
of  the  AFHRL  report  have  made  an  effort  to  identify  them  for 
the  reader  and  to  provide  sufficient  information  so  that 
their  findings  can  be  combined  with  results  obtained  else- 
where. By  providing  ample  description  of  the  constraints 
under  which  the  data  were  collected,  the  authors  enable 
interested  readers  to  draw  upon  these  results  and  others  to 
state  generalizations  and  hypotheses  for  further  investiga- 
t ion . 

Although  no  explicit  rationale  is  given  for  includ- 
ing the  summary  of  CBE  effectiveness  studies,  it  is  apparent 
that  one  purpose  is  to  highlight  an  issue  in  evaluation 
design  found  in  several  of  the  studies  cited.  Namely,  when 
CBE  or  other  self-paced  media  are  compared  with  conventional 
classroom  presentations,  the  time  savings  typically  enjoyed 
by  the  CBE  groups  can  not  be  attributed  entirely  to  the 
effects  of  CBE.  Such  savings  might  be  due  merely  to  the 
fact  that  self-pacing  or  group-pacing  has  been  employed 
rather  than  to  any  unique  CBE  characteristics.  Since  CBE  was 
to  be  compared  with  a conventional  lock-stepped  presentation 
format  in  the  Chanute  service  test,  the  question  of  how  to 
isolate  CBE's  unique  contribution  was  an  important  one  in  the 
selection  of  an  evaluation  paradigm.  Two  studies  were  cited 
in  which  this  problem  was  solved  by  comparing  the  CBE  mater- 
ials with  comparable  materials  presented  by  other  self-paced 
media.  This  type  of  control  was  attempted  for  a few  lessons 
in  the  Chanute  service  test  (see  section  on  PLATO  system 
effectiveness  in  Chapter  III),  but  the  lack  of  non-CBE  (i.e., 
off-line)  self-paced  materials  made  it  impossible  to  imple- 
ment on  a larger  scale. 
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Another  apparent  reason  for  citing  CBE  effectiveness 
studies  was  to  summarize  their  findings  as  background  for 
the  results  obtained  in  the  Chanute  service  test.  A study 
of  particular  interest  was  that  conducted  by  the  LI.  S.  Army 
Ordnance  Center  and  School  (Dept  of  the  Army,  1975).  It 
.nvolved  a comparison  of  the  time  required  to  achieve  a pre- 
set criterion  of  mastery  for  each  of  a series  of  CBE  lessons 
and  for  a parallel  series  of  lessons  presented  on  conven- 
tional self-paced  media.  The  project  staff  reported  time 
savings  in  favor  of  CBE  in  most  of  the  lesson-by-lesson  com- 
parisons (Dept  of  the  Army,  1975,  p.  C-4),  but  an  independent 
examination  of  the  data  and  analysis  procedures  revealed 
difficulties  that  make  some  of  the  results  equivocal  (Avner 
and  Misselt,  1977).  Avner  and  Misselt  do  conclude,  however, 
that  the  Army  project's  conclusions  with  regard  to  educa- 
tional effectiveness  of  CBE  are  generally  supported  by  the 
available  data.  Related  reports  by  Himwich  (1977b)  and 
Call-Himwich  (1977)  discuss  the  implementation  and  management 
of  the  Army  project  and  give  a critical  appraisal  of  the  CBE 
lessons  that  were  developed. 
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Overall  Methodology 

Cond itions . The  description  of  the  training  conditions 
or  "treatments"  which  were  established  for  comparison  pur- 
poses is  factually  correct  although  it  may  be  too  brief  to 
give  a clear  understanding  of  the  way  in  which  they  were 
implemented.  It  is  important  to  emphasize  that  the  ISD 
revision  of  the  four  Special  Purpose  Vehicle  courses  (i.e., 
the  shreds)  included  not  only  the  first  four  blocks  common  to 
all  shreds  (termed  the  common  course  segment)  but  also  the 
remaining  blocks  of  each  shred.  The  revisions  to  the  common 
course  segment  were  more  extensive  in  that  all  lectures  were 
replaced  by  PLATO  lessons  and  other  self-paced  media,  but  the 
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contents  of  the  "shred  areas"  (i.e.,  the  blocks  of  each 
course  following  the  common  course  segment)  were  also  changed 
to  reflect  the  revised  course  goals  (established  by  the  ISD 
team  in  their  analysis  of  job  requirements) . The  only  use  of 
PLATO  in  the  shred  areas  was  in  the  administration  and  scor- 
ing of  block  exams;  no  CBE  lessons  were  used. 

The  revision  of  the  shred  areas  was  conducted  concur- 
rently with  that  of  the  common  course  segment.  The  B shred 
revision  was  completed  first  (note  that  according  to  Figure  2 
in  Chapter  II,  the  B shred  had  the  fewest  blocks)  so  opera- 
tional trials  of  the  revised  version  could  begin  15  Jan  1975. 
The  D shred  revision  was  the  next  to  be  completed  with  the 
first  trial  class  starting  on  13  March  1975.  The  C shred  was 
ready  for  its  first  class  by  22  May  1975,  while  the  A shred 
(the  longest  according  to  Figure  2)  was  not  completely 
revised  until  13  June  1975. 

It  was  this  staggered  completion  schedule  that  defined 
the  treatment  conditions.  Because  the  A shred  revision  was 
the  last  to  be  completed,  classes  entering  that  course 
between  January  and  June  of  1975  were  given  the  conventional 
version  of  the  curriculum.  These  classes  were  the  only  ones 
assigned  to  the  NP  (conventional)  condition.  Classes  enter- 
ing the  C and  D shreds  between  January  1975  and  the  time  that 
the  shred  area  revisions  were  complete  were  assigned  to  the 
CP  condition.  The  PB  condition  was  made  up  of  the  B shred 
classes  and  those  classes  entering  the  other  shreds  after 
their  respective  revisions  had  been  completed.  The  BL  con- 
dition included  classes  from  all  four  shreds  which,  of 
course,  received  conventional  training  during  the  common 
course  segment  as  well  as  the  shred  areas. 

Because  of  the  confounding  between  shred  assignment  and 
condition  (e.g.,  NP  consisted  only  of  A shred  students), 
interpretation  of  the  results  rests  on  the  assumption  that 
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there  were  no  consistent  differences  among  the  shreds  in 
terms  of  student  aptitude,  teaching  ability  of  instructors, 
and  other  factors  likely  to  affect  performance.  The  fact 
that  the  treatment  conditions  were  defined  in  this  way, 
rather  than  by  random  assignment  or  some  other  form  of  con- 
trol, is  a threat  to  the  internal  validity  of  the  design 
(Campbell  & Stanley,  1963).  However,  it  exemplifies  the 
kind  of  compromise  between  the  requirements  of  research  and 
those  of  operational  and  logistic  concerns  that,  unfortun- 
ately, must  be  made  when  an  operational  orientation  is  para- 
mount . 

Student  population.  As  indicated  by  the  authors,  the 
differing  sample  sizes  were  due  entirely  to  the  staggered 
implementation  of  the  ISD  revisions  to  the  various  shreds. 
This  is  one  area  in  which  the  constraints  placed  on  the 
evaluation  by  the  operational  orientation  of  the  service  test 
were  particularly  severe.  If  it  had  been  possible  for  the 
evaluators  to  control  the  rate  of  implementation,  more  uni- 
form sample  sizes  could  have  been  obtained. 

In  general,  overall  sample  sizes  are  judged  to  be  suf- 
ficient to  provide  statistical  tests  of  sufficient  power  to 
detect  the  major  effects  likely  to  be  of  interest  in  instruc- 
tional settings.  Detailed  power  analyses  are  given  for  indi- 
vidual comparisons  later  in  this  critique.  The  probability 
of  detecting  true  differences  of  at  least  one  standard  devi- 
ation between  means  for  the  major  groups  (PB,CP,NP,  and  BL) 
was  at  least  .95  (for  an  alpha  level  of  .05).  The  prrobabil- 
ity  of  detecting  smaller  differences  likely  to  be  of  research 
interest  is,  however,  quite  small  (no  higher  than  .56  for  any 
comparison  involving  the  NP  group  in  detecting  a true  dif- 
ference of  at  least  .5  standard  deviations,  again  with  alpha 
of  .05) . 


III.  INSTRUCTIONAL  EFFECTIVENESS 


When  the  agencies  most  closely  involved  with  the  service 
test  were  surveyed  regarding  their  priorities  for  the  evalu- 
ation, the  question  of  instructional  effectiveness  was  seen 
to  be  of  prime  importance.  It  is  not  an  easy  question  to 
answer — given  the  constraints  arising  from  the  "operational" 
orientation  of  Phase  II--but  those  persons  responsible  for 
the  evaluation  have  demonstrated  considerable  resourcefulness 
in  dealing  with  the  problem.  Chapter  III  gives  a well  organ- 
ized and  informative  account  of  the  various  data  collection 
procedures  and  analyses  used  in  addressing  the  instructional 
effectiveness  issue. 

Special  Purpose  Vehicle  Course 

Special  purpose  vehicle  course  procedures.  This  section 
gives  a concise  description  of  the  variables  measured  and 
the  procedures  used  in  data  collection.  No  further  comment 
is  needed. 

Block  exam  scores.  As  mentioned  in  an  earlier  section 
of  this  critique,  the  block  exams  in  the  common  course  seg- 
ment and  the  shred  areas  had  to  be  revised  to  reflect  scope 
and  depth-of-content  changes  resulting  from  the  ISD  effort. 
Once  the  revised  courses  were  fully  implemented,  all  students 
received  the  same  instruction  during  the  four  blocks  of  the 
common  course  segment  and  were  tested  using  the  revised  block 
exams.  During  the  period  when  the  evaluation  data  were  col- 
lected, however,  only  the  students  in  the  PB  condition  were 
given  the  revised  exams — students  in  the  NP  and  CP  conditions 
took  the  original  (unrevised)  versions.  It  is  for  this  rea- 
son that  the  PB  condition  could  not  be  included  in  the  analy- 
ses of  variance  summarized  in  Table  3- 
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A comment  is  in  order  regarding  the  use  of  ANOVA  to 
perform  separate  univariate  analyses  for  each  of  the  four 
block  exams.  Unless  the  variables  (e.g.,  the  block  exams) 
are  independent  of  each  other,  it  is  inadvisable  to  conduct 
separate  univariate  hypotheses  tests  for  each  of  several  var- 
iables measured  on  the  same  set  of  subjects.  Another  pos- 
sible analysis  in  such  cases  is  multivariate  analysis  of  var- 
iance (MANOVA)  (see  Tatsuoka,  1971).  If  the  comparisons  had 
been  independent  and  if  group  sizes  had  been  roughly  equal 
(with  the  same  total  sample  size),  the  tests  would  have  had 
the  power  to  detect  a moderate  effect  (one  accounting  for 
5.88%  of  population  variance)  with  probability  of  at  least 
.96. 

A final  comment  on  the  reporting  of  the  block  exam 
results  is  that  no  information  regarding  the  reliabilities 
of  these  exams  is  provided.  At  a minimum,  it  would  have  been 
useful  to  have  reported  the  number  of  items  on  each  exam 
along  with  the  means  and  standard  deviations  given  in  Table 
2.  With  this  additional  information,  it  would  be  possible  to 
estimate  test  reliability  using  the  KR-21  estimator. 

Course  completion  time.  The  rate  at  which  students  com- 
plete the  course  is  an  important  variable  in  assessing  the 
efficiency  of  the  ISD  revisions,  but  it  does  not  directly 
measure  their  effectiveness.  Presumably  it  is  an  indirect 
measure,  however,  because  students  would  not  progress  from 
block  to  block  if  their  instruction  were  so  ineffective  as  to 
prevent  them  from  passing  the  block  exams. 

Time  savings  due  to  the  course  revisions  were  computed 
by  comparing  mean  completion  times  under  the  revised  course 
with  the  baseline  completion  times  under  the  original  course. 
Comparisons  were  made  within  each  shred  for  the  common  course 
segment  (which  included  PLATO  instruction)  and  for  the  shred 
taken  as  a whole  (i.e.,  including  the  shred  area  blocks). 
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It  is  apparent  from  Table  4 in  the  AFHRL  report  that  time  was 
saved  in  each  of  the  common  course  segment  blocks,  but  that 
some  shred  area  blocks  took  longer  to  complete  under  the  ISO 
revision  than  under  the  baseline  version.  This  latter  obser- 
vation reflects  the  ISO  redistribution  of  course  content  in 
some  of  the  shred  area  blocks--some  blocks  were  shortened 
while  others  were  lengthened. 

Table  5 in  the  report  presents  the  mean  hours  saved  per 
block  and  the  percentage  of  the  baseline  completion  time  that 
was  saved  in  each  shred  during  the  common  course  segment.  It 
is  stated  in  the  text  of  the  report  that  28.5%  of  the  common 
course  segment  baseline  time  was  saved  in  the  ISD  revision. 

It  should  be  recalled  from  earlier  discussion  that  some  of 
this  time  savings  is  due  to  moving  some  content  items  to 
Block  X (home  study)  and  much  of  the  remainder  is  due  to  the 
introduction  of  group-pacing  to  replace  lock-step  pacing. 

It  may  also  be  that  the  use  of  PLATO  contributed  to  this 
overall  time  savings  by  saving  some  time  above  and  beyond 
that  expected  by  delivering  the  same  instruction  with  other 
self-paced  media.  Additional  discussion  of  this  point  is 
given  in  Chapter  IV. 

Eliminations . It  was  reasonable  to  plan  to  examine  the 
frequency  of  academic  eliminations  in  the  various  treatment 
conditions,  but,  as  stated,  there  were  so  few  eliminations 
during  the  evaluation  period  that  analysis  was  unnecessary. 

Washbacks . No  further  comment  is  needed. 

Absenteeism . The  number  of  absences  is  another  of  the 
course  parameters  which  is  easily  measured  and  which  permits 
comparisons  among  the  various  conditions.  Its  relevance  to 
the  question  of  instructional  effectiveness  is  not  clear, 
however,  as  it  seems  to  be  more  of  a measure  of  motivation 
than  a measure  of  learning.  As  such,  it  might  more  appropri- 
ately have  been  discussed  in  Chapter  V along  with  the  other 
indices  of  instructional  impact. 
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The  absentee  data  were  analyzed  by  the  Chi  square  test 
for  independence.  The  total  number  of  absences  and  the  mean 
number  of  non-absences  were  compared  across  conditions  in  a 
series  of  two-way  classification  tables.  Unfortunately,  the 
discussion  of  how  the  mean  number  of  non-absences  was  com- 
puted is  given  in  too  little  detail  for  us  to  evaluate  its 
logic.  Without  additional  information,  we  are  unable  to  ver- 
ify the  correctness  of  the  procedure  or  the  accuracy  of  the 
results . 

Regardless  of  the  outcomes  of  the  analyses  of  absentee 
frequencies,  the  supplemental  AMOVA  on  number  of  hours  absent 
seems  to  have  been  conducted  appropriately.  No  significant 
differences  among  the  conditions  were  found  on  this  measure. 

Special  individualized  assistance.  While  the  extended 
median  test  is  applicable  in  the  case  described,  use  of 
Scheffi's  multiple-comparisons  approach  would  have  permitted 
the  post  hoc  conclusion  (that  the  CP  students  were  given  more 
assistance)  to  have  been  made  explicitly  rather  than  as  a 
qualitative  observation. 

Field  evaluation.  The  first  observation  likely  to  be 
made  in  a careful  reading  of  this  section  is  that  there  is  a 
discrepancy  between  the  number  of  NP  students  purportedly 
studied  in  the  field  evaluation  (28)  and  the  sample  size 
reported  for  earlier  analyses  (21).  It  is  our  understanding 
that  several  students  who  were  trained  under  the  NP  condition 
were  omitted  from  earlier  analyses  because  of  incomplete 
data.  Missing  data  on  other  variables  should  have  no  bearing 
on  field  evaluation  analyses,  however,  so  it  seems  appropri- 
ate that  these  students  have  been  included  in  the  NP  sample. 

The  discussion  of  the  analyses  of  the  field  evaluation 
ratings  data  is  fairly  complete,  and  the  analyses  seem  appro- 
priate for  the  available  data.  The  major  finding  was  that 
there  were  no  significant  differences  among  the  conditions 
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with  regard  to  the  overall  performance  ratings  and  a derived 
score  based  on  the  ratings  for  the  33  individual  target 
tasks.  A supplemental  analysis  showed  that  supervisor  rat- 
ings on  four  of  these  tasks  were  significantly  related  to  the 
overall  performance  rating  for  those  students  whose  duty 
assignments  were  at  the  apprentice  level. 

Discussion  of  course  effectiveness.  The  discussion  in 
this  section  addresses  the  question  of  whether  or  not  the  two 
PLATO-using  treatment  conditions  (PB  and  CP)  were  instruc- 
tionally  effective.  The  premise  on  which  the  arguments  seem 
to  be  based  is  that  the  PB  and  CP  conditions  should  be  con- 
sidered to  be  "effective"  if  they  fare  ^ well  as  the  NP  and 
BL  conditions  on  the  various  indices  of  course  effectiveness 
that  were  reported.  Accepting  this  premise  assumes  that  such 
measures  as  number  of  washbacks,  rate  of  absenteeism,  hours 
of  SIA  time,  and  instructor  ratings  in  the  field  are  adequate 
indicators  of  effectiveness.  While  most  of  these  measures 
are  valid  for  the  desired  purpose,  many  of  them  are  insensi- 
tive (in  comparison  to  tests)  and  are  of  limited  value  in 
discriminating  among  treatment  conditions.  Because  the  PB 
common  course  segment  block  exams  were  different  from  those 
taken  by  the  other  groups,  they  could  not  be  used  for  compar- 
ative purposes.  Fortunately,  the  CP  group  took  the  same 
block  exams  as  did  the  NP  and  BL  groups  so  that  a more  sensi- 
tive measure  of  CP  condition  effectiveness  was  available. 

Inferring  from  the  pattern  of  results  on  the  set  of 
relatively  insensitive  indicators  described  above,  the 
authors  concluded  that  the  PB  condition  was  at  least  as 
effective  as  the  NP  and  BL  conditions.  Likewise,  they  con- 
cluded that  the  CP  condition  was  "effecti ve"--according  to 
the  criterion  described  above--because  it  fared  as  well  on 
the  various  measures  (including  the  block  exams)  as  did  the 
NP  and  BL  conditions. 


Although  the  discussion  in  this  section  of  the  report 
appears  to  us  to  be  accurate,  it  suffers  from  lack  of  clar- 
ity. Furthermore,  the  arguments  do  not  seem  convincing.  We 
believe  that  much  of  this  problem  stems  from  the  fact  that 
the  authors  chose  not  to  present  the  results  found  with 
another  measure — the  special  topical  test.  Because  we  feel 
it  provides  additional  insight  into  the  question  of  course 
effectiveness,  we  wish  to  supplement  the  AFHRL  report  by  dis- 
cussing this  test  and  the  results  obtained  with  its  use.  We 
have  had  access  to  this  information  due  to  our  role  as  data 
analysts  in  assisting  with  the  service  test. 

A special  topical  test  was  constructed  by  the  service 
test  evaluation  team  for  the  purpose  of  attempting  to  isolate 
the  effects  of  the  PLATO  system  from  those  of  the  other 
changes  made  during  the  ISD  redesign  of  the  curriculum.  This 
topical  test--mentioned  briefly  in  the  report  under  the  head- 
ing "PLATO  System  Ef fectiveness"--consisted  of  30  items  test- 
ing material  covered  in  the  CBE  lessons  and  20  items  over 
topics  presented  by  other  media.  The  original  intent  of  the 
evaluators  was  to  compare  the  various  treatment  groups  on  the 
basis  of  scores  for  each  of  these  two  sets  of  items  (i.e.,  a 
PLATO  item  score  and  a non-PLATO  item  score).  Presumably,  it 
would  be  possible  to  determine  from  these  comparisons  whether 
PLATO  had  effects  above  and  beyond  those  of  the  other  "ISD 
influences".  Because  of  excessive  item  difficulty  and  other 
technical  problems,  however,  the  PLATO  item  and  non-PLATO 
item  sub-scores  were  highly  unreliable  (coefficient  alpha 
reliabilities  of  .55  and  .38,  respectively)  and  these  results 
were  not  reported  (see  section  on  "PLATO  System 
Effectiveness" ) . 

While  it  is  appropriate  that  the  sub-score  results  were 
omitted  from  the  report,  a second  set  of  analyses  based  on 
the  somewhat  more  reliable  total  scores  (i.e.,  the  sum  of  the 
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PLATO  and  non-PLATO  item  sub-scores)  gives  valuable  informa- 
tion on  the  question  of  course  effectiveness,  the  question 
addressed  in  this  section.  The  reliability  of  the  total 
scores  falls  short  of  what  would  be  desirable  for  most  pur- 
poses (coefficient  alpha  = .65),  but,  if  interpreted  cau- 
tiously, the  results  can  be  used  as  a check  on  findings 
obtained  with  other  measures. 

The  special  topical  test  was  actually  administered  three 
times  during  the  flow  of  the  course — once  as  a pretest,  once 
at  the  conclusion  of  the  common  course  segment  (posttest  1), 
and  once  at  the  end  of  instruction  in  the  shred  areas  (post- 
test 2).  It  had  been  planned  by  the  evaluation  team  to  com- 
pare the  treatment  groups  at  each  of  these  three  points  in 
time,  but  for  the  present  purpose  it  is  sufficient  to  con- 
sider only  the  pretest  and  posttest  1. 

There  were  no  significant  differences  among  the  groups 
on  the  pretest  (F(2, 145)= .434,  p=.649),  an  indication  that 
they  were  homogeneous  with  regard  to  initial  knowledge  of  the 
topics  covered.  As  can  be  seen  from  Table  1 (in  this  cri- 
tique) , the  MP  group  had  the  smallest  mean  performance  on 
posttest  1. 


Table  1 

Group  Means  and  Standard  Deviations 
on  Posttest  1 Total  Scores 


Group 

Mean 

Stand  ard 
Deviation 

n 

FB 

28.111 

4.696 

81 

CP 

28.727 

5.398 

44 

NP 

23.565 

4.998 

23 
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A one-way  fixed-effects  ANOVA  on  posttest  1 total  scores 
is  summarized  in  Table  2.  The  effect  underlying  the  signifi- 
cant F statistic  accounts  for  11.2%  of  total  sample  variance. 
The  power  of  this  test  to  detect  an  effect  of  moderate  size 
(accounting  for  5.88%  of  population  variance)  is  only  about 
.76,  so  a smaller  effect  might  easily  have  missed  detection. 
Post  hoc  analyses  using  the  Scheffft  method  show  the  PB  group 
performance  to  be  significantly  higher  than  that  of  the  NP 
group  (p  = .001).  The  CP  group  also  had  significantly  higher 
scores  than  the  NP  students  (p  < .0005),  but  the  difference 
between  the  PB  and  CP  groups  was  not  significant  (p  = .803) 
and  is  most  likely  due  to  chance  effects. 


Table  2 


Summary  of  ANOVA  on 

Posttest 

1 Total 

Scores 

Source  df 

MS 

F 

P 

trtmnt  2 

225.741 

9.177 

.0002 

error  145 

24.598 

— 

— 

The  special  topical  test  results  seem  to  indicate  that 
the  ISD-revised  course  taken  by  the  PB  students  was  more 
effective  than  the  original  instruction  on  these  topics.  The 
fact  that  the  CP  group  also  fared  well  relative  to  the  NP 
group  is  due  to  the  combined  effects  of  the  CBE  materials  and 
supplementary  discussions/review  provided  by  the  instructors. 
It  is  interesting  that  this  supplemental  instruction  did  not 
raise  the  scores  of  the  CP  students  significantly  above  those 
of  the  PB  group  even  though  more  training  time  was  expended 
(see  Table  4 in  the  report).  It  should  be  remembered  that 
coefficient  alpha  for  the  special  topical  test  was  only  .65, 
but,  on  the  other  hand,  the  major  effect  of  unreliability  in 
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ANOVA  is  to  obscure  the  differences  amonc  treatments  by 
reducing  statistical  power  (for  a discussion  of  this  point, 
see  Cochran,  1968;  Misselt,  1977).  It  is  important  that  sig- 
nificant differences  were  found  in  spite  of  such  unreliabil- 
ity. Although  there  may  be  other  confounding  factors  (e.g., 
differing  levels  of  experience/effectiveness  among  intructors 
for  the  various  conditions)  which  could  account  for  these 
results,  they  seem  to  add  direct  support  for  the  conclusion 
drawn  in  the  report  itself  that  the  PLATO-using  conditions 
were  "at  least"  as  instructionally  effective  as  the  non- 
PLATO  versions. 

In  addition  to  the  authors'  decision  to  base  their  dis- 
cussion of  course  effectiveness  on  what  we  consider  to  be 
relatively  insensitive  measures,  another  problem  detracting 
from  the  clarity  of  the  presentation  is  their  tendency  to 
intersperse  discussion  of  "efficiency"  issues  within  the 
context  of  the  "effectiveness"  question.  It  is  desirable  to 
compare  the  efficiency  of  the  various  treatment  conditions 
but  it  is  important  not  to  confuse  measures  of  efficiency 
(e.g.,  the  completion  time  required  in  the  various  blocks) 
with  those  of  effectiveness.  With  regard  to  instructional 
efficiency,  there  is  ample  evidence  that  the  revisions  did 
reduce  training  time.  This,  after  all,  was  the  major  goal  of 
the  ISD  team.  They  did  not  set  out  to  teach  more  content  or 
teach  to  a higher  degree  of  mastery.  Instead,  they  attempted 
to  make  the  common  course  segment  instruction  more  efficient. 
They  apparently  were  successful  in  this  effort. 

PLATO  System  Effectiveness 

Procedures . The  evaluation  team  attempted  two  proce- 
dures for  assessing  the  effectiveness  of  the  PLATO  system  (as 
used  at  Chanute)  apart  from  the  effects  of  other  important 
variables  in  the  environment.  The  first  of  these,  mentioned 
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only  briefly  in  the  report,  was  a plan  to  use  the  sub-scores 
on  the  special  topical  test  to  look  for  differences  in  learn- 
ing between  the  treatment  groups  on  two  sets  of  topics--those 
covered  by  PLATO  lessons  and  those  taught  using  more  conven- 
tional media.  Any  differences  among  the  groups  on  the  non- 
PLATO  item  sub-scores  presumably  were  to  be  taken  as  evidence 
of  differential  effectiveness  between  the  revisions  and  the 
original  instruction  on  these  topics.  If,  for  example,  the 
PB  group  were  to  do  no  better  than  the  NP  group  on  the  non- 
PLATO  item  subtest,  it  would  have  been  an  indication  that  the 
ISD  procedural  changes  inherent  in  the  PB  condition  had  no 
special  effect.  If,  meanwhile,  the  PB  group  had  done  signi- 
ficantly better  than  the  NP  group  on  the  PLATO  item  subtest, 
it  would  have  been  taken  as  evidence  that  Chanute's  use  of 
PLATO  had  effects  above  and  beyond  those  of  the  remaining  ISD 
influences.  This  plan  was  not  implemented  because  of  the  low 
reliability  of  the  sub-scores  (see  discussion  in  the  previous 
section  of  this  critique).  However,  even  if  the  tests  had 
been  more  carefully  developed  so  as  to  have  adequate  relia- 
bility, it  is  not  immediately  clear  that  it  would  have  been 
possible  to  separate  the  effects  of  PLATO  from  those  of  the 
other  environmental  changes  using  the  plan  outlined  above. 

The  second  procedure  used  in  attempting  to  determine  the 
effects  of  the  PLATO  system  (i.e.,  while  controlling  other 
variables)  was  a special  comparison  between  a group  studying 
four  of  the  PLATO  lessons  and  a group  given  four  programmed 
texts  over  the  same  set  of  topics.  Because  the  programmed 
texts  used  electrostatic  copies  of  screen  displays  (or 
frames)  from  the  PLATO  lessons,  the  extraneous  effects  of 
subject  matter  and  lesson  strategy  were  controlled  (Salomon  & 
Clark,  1977).  The  comparisons  were  based  on  end-of-lesson 
test  scores  and  the  time  required  to  complete  the  lessons. 

The  students  studying  the  programmed  text  versions  were  also 
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asked  to  state  their  preference  between  programmed  texts  and 
PLATO  (they  had  used  PLATO  to  study  other  topics). 

The  discussion  of  these  procedures  is  clear  and  concise,  but 
it  would  have  been  useful  to  add  some  description  of  the 
data  collection  procedures  and  some  assessment  of  the 
reliability  of  the  measures  used  (including  the  completion 
time  measurements) . 

Results . No  significant  differences  in  lesson  test  per- 
formance were  found  between  the  PLATO  and  Programmed  Text 
( PT ) comparison  groups.  One  concern  to  be  noted  is  that  the 
tests  used  were  originally  designed  to  be  criterion  refer- 
enced tests.  Because  criterion  referenced  tests  tend  to  give 
a "ceiling"  effect  and  have  low  variances,  they  are  generally 
not  very  suitable  for  comparison  purposes  (Popham,  1971). 

Even  so,  a more  senstitive  measure  could  have  been  obtained 
by  summing  the  item  scores  on  the  four  lesson  tests  to  give 
a single  overall  score. 

In  the  absence  of  information  about  the  reliability 
of  the  completion  time  data,  little  can  be  said  about  the 
results  obtained.  It  is  important,  of  course,  that  the 
PLATO  group  had  fewer  individual  failures  and  that  the  PT 
group  indicated  a preference  to  studying  lessons  on  PLATO. 

Discussion  of  comparison  findings.  The  discussion  in 
this  section  is  ca»'efully  reasoned  and  accurate.  The  conclu- 
sion that  the  PLATO  lessons  were  more  instructionally  effec- 
tive than  their  PT  counterparts  seems  justified  by  the  data 
presented.  The  caution  about  overgeneralization  of  these 
results  is  an  appropriate  one,  however. 

PLATO  Courseware 

Procedures . As  indicated  by  the  report  authors,  there 
were  insufficient  resources  to  develop  parallel  non-PLATO 
instruction  on  each  of  the  topics  targeted  for  CBE.  Since  no 


89 


other  lesson  materials  covering  the  appropriate  content  were 
available,  it  was  impossible  to  do  a comparative  evaluation 
on  a lesson-by-lesson  basis.  The  alternative  was  to  deter- 
mine the  extent  to  which  each  lesson  was  capable  of  bringing 
students  to  a pre-determined  level  of  performance. 

Once  a lesson  was  shown  to  have  been  effective  instruc- 
tion for  a suitable  validation  sample,  it  was  said  to  be 
"validated".  The  proportion  of  students  passing  each  les- 
son's Master  Validation  Exam  (MVE)  was  monitored  continuously 
during  the  evaluation  of  Phase  II.  When  a validation  attempt 
failed  (i.e.,  more  than  10%  of  a 20-30  student  sample  failed 
to  pass  the  MVE),  the  lesson  was  revised  in  the  hope  of  mak- 
ing it  more  effective.  Once  a validation  attempt  was  suc- 
cessful, no  more  revisions  were  made  (this  procedure  was 
based  on  the  premise  that  lessons  should  be  developed  as 
"lean"  as  possible  initially,  then  making  only  those  addi- 
tions which  are  necessary  for  them  to  validate). 

A check  on  the  "appropriateness"  of  the  decision  to  con- 
sider a lesson  as  having  validated  was  made  by  monitoring 
"post  validation"  failure  rates.  If  the  failure  rate  was  too 
high  (greater  than  20%),  the  lesson  was  to  be  revised  and 
subjected  to  new  validation  trials.  Failure  rates  between  10 
and  20%  were  an  indication  that  a lesson  should  be  monitored 
carefully.  Only  if  no  more  than  10%  of  the  post  validation 
students  failed  the  MVE  was  the  lesson  considered  to  be 
instructionally  stable. 

Results . The  fact  that  four  lessons  definitely  needed 
revision  and  eight  others  may  have  needed  to  be  reworked 
indicates  that  the  original  validation  criterion  was  too 
lenient.  That  a systematic  validation  procedure  was  followed 
at  all  is  certainly  commendable,  but  the  procedure  used  seems 
arbitrary  and  intuitively  based.  A more  scientific  valida- 
tion procedure  has  been  developed  by  Tatsuoka  (1977).  It 


applies  a Bayesian  statistical  model  to  the  data  from  a vali- 
dation attempt  and  derives  an  estimate  of  the  probability 
that  the  lesson  will  "remain”  validated  in  the  population 
from  which  the  initial  sample  was  drawn.  It  was  possible 
with  this  procedure  to  predict  which  lessons  would  be  most 
likely  to  ”de-validate"  over  time.  The  data  given  in  the 
AFHRL  report  confirm  these  predictions.  A comparison  of 
Chanute’s  validation  approach  and  the  proposed  alternative  is 
given  in  the  report  by  Tatsuoka  (1977).  Tatsuoka's  report 
also  discusses  the  shortcomings  of  defining  "validation"  in 
terms  of  test  scores;  a poorly-constructed  test  may  not  be  an 
adequate  instrument  for  judging  the  effectiveness  of  a 
lesson.  There  is  a need  for  establishing  a better  validation 
criterion . 

The  analysis  of  on-line  time  data  is  probably  more  rele- 
vant to  the  topic  of  instructional  efficiency  than  it  is  to 
instructional  effectiveness.  In  any  event,  however,  the 
statement  that  further  time  savings  could  be  realized  from  a 
self-paced  course  structure  is  quite  justified. 

Discussion  of  lesson  performance  r?sults.  The  discus- 
sion in  this  section  is  well-stated  and  complete.  No  further 
comments  are  needed. 

Instructional  Effectiveness  Conclusions 

The  major  conclusion  regarding  effectiveness  is  that 
the  PLATO  lessons  and  other  materials  and  procedures  intro- 
duced in  the  new  course  led  to  adequate  student  learning.  We 
believe  the  available  data  supports  this  conclusion. 

A second  conclusion  is  that  the  revised  course  is  more 
efficient  because  it  requires  less  training  time  to  complete. 
Part  of  this  time  savings  may  be  due  to  such  things  as  depth- 
of-instruction  changes,  moving  some  of  the  material  from  the 
common  course  segment  to  the  shred  areas  or  to  Block  X (i.e.. 
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home  study),  and  the  change  to  a group-paced  format.  It  is 
not  possible,  therefore,  to  attribute  all  of  the  time  savings 
to  more  efficient  teaching  strategies  or  the  standardization 
of  instruction,  but  it  is  likely  that  these  account  for  some 
of  the  efficiency  advantages. 


C ; ing  true  operating  costs  in  realistic  operating  environments. 


Actual  instructional  settings  are  rarely  composed  of  indepen- 
dent accounting  units.  Thus  it  is  usually  necessary  to  infer 
costs  as  some  fraction  of  a total  which  includes  the  expenses 
of  unrelated  activities.  Experimental  operations  also  are 
rarely  able  to  make  use  of  the  savings  that  would  accrue  from 
saturation-use  of  delivery  facilities.  Under  experimental 
conditions,  there  is  a need  to  provide  sufficient  unscheduled 
resources  to  permit  exploitation  of  research  opportunities 
which  are  discovered  during  a project.  In  a "production" 
setting,  all  available  resources  would  be  fully  utilized  with 
the  result  that  costs  would  be  shared  by  more  users,  and 
cost-per-unit-instruction  would  be  decreased. 

Instead  of  following  the  headings  and  format  of  Chapter 
IV,  in  which  all  costs  are  separately  categorized,  this  sec- 
tion of  the  critique  converts  presentation  costs  into  a sin- 
gle form  for  ease  in  gaining  an  overview. 

Using  industry  standards,  costs  may  be  determined  on  an 
annual  basis  by  assuming  electronics  hardware  to  have  a seven 
year  life  span,  classroom  improvements  to  have  a ten  year 
life  span,  and  interest  costs  to  the  Government  to  be  1%.  We 
then  find  the  total  capital  expenditures  of  $191,688  have  an 
imputed  annual  cost  of  $35,568  per  year,  the  $4,803  spent  for 
facilities  preparation  have  an  annual  cost  of  $684,  and  oper- 
ational costs  are  $89,933,  for  a total  annual  cost  of  about 
$126,185.  These  figures  may  also  be  interpreted  as  costs  per 
terminal  of  $4,206  per  year  (with  30  terminals)  or  as  costs 
per  terminal  hour  of  $3*85  (using  the  figure  of  1092  hours 
usage  per  terminal  per  year  cited  in  Table  20  of  the  report) . 


Since  less  than  of  the  annual  cost  of  terminal  oper- 
ation is  a function  of  operating  time  (e.g.,  the  cost  of 
electricity  for  operating  the  terminal),  the  cost  per  hour  of 
a terminal  could  be  lowered  substantially  by  higher  utiliza- 
tion rates.  The  average  weekly  use  of  the  terminals  was  less 
than  25  hours,  a figure  that  could  have  been  doubled  with 
moderate  ease  in  a full-scale  training  situation.  The  hourly 
terminal  cost  is  of  interest  because  it  is  this  figure  which 
determines  whether  a savings  in  training  time  from  use  of  CBE 
is  sufficient  to  make  use  of  CBE  cost-effective. 

Based  on  the  estimated  student  time  savings  of  1 1/3 
training  days  (or  $64)  cited  in  the  section  on  "real  sav- 
ings", time  savings  would  pay  for  only  about  16.6  hours  of 
CBE  time  when  terminals  were  used  for  no  more  than  25  hours 
per  week.  If  terminal  usage  were  doubled,  as  suggested  by 
the  AFHRL  report,  the  hourly  rate  per  terminal  would  decrease 
and  the  observed  time  savings  would  have  paid  for  over  33 
hours  of  terminal  time.  Since  these  times  bracket  the 
observed  student  on-line  times  in  the  course,  it  appears  that 
CBE  was  close  to  being  as  cost-effective  in  this  situation  as 
programmed  texts.  A complete  analysis  of  such  a comparison 
would,  of  course,  be  possible  only  if  data  for  larger-scale 
implementations  of  both  approaches  were  available  for  this 
situation.  Also  needed  would  be  more  detail  on  indirect 
costs  associated  with  use  of  CBE  and  alternative  media  at 
Chanute.  Unfortunately,  the  basis  for  many  of  the  cost  fig- 
ures reported  in  this  chapter  is  not  given  in  sufficient 
detail  to  go  beyond  the  analysis  given  by  the  authors.  Given 
such  added  detail,  potential  savings  in  instructor  time  and 
marginal  benefits  in  increased  student-instructor  satisfac- 
tion could  also  be  evaluated.  It  is  possible  that  the  dis- 
tributed instructor  time-savings  described  by  the  authors 
could  be  applied  to  a heavier  student  load  in  some  situa- 
tions, for  example. 
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In  summary,  if  the  AFHRL  estimate  of  a 10%  time  savings 
due  to  CBE  is  correct,  it  appears  that  the  experimental 
implementation  was  close  to  being  cost-effective  and  that  the 
more  complete  resource  utilization  possible  in  a full-scale 
implementation  might  be  cost-effective  even  with  currently 
available  hardware. 


V.  INSTRUCTIONAL  IMPACT 


Chapter  V gives  an  extensive  and  careful  discussion  of 
several  issues  relating  to  the  incorporabil ity  of  PLATO  in 
the  Air  Force  technical  training  environment.  It  is  a very 
informative  and  well-organized  account.  However,  for  the 
sake  of  convenience,  our  discussion  uses  a system  of  headings 
and  sub-headings  which  is  slightly  less  detailed  than  that 
used  in  the  chapter. 

Introduction 

The  rationale  for  including  the  study  of  student  and 
instructor  attitudes  in  the  evaluation  is  quite  valid.  The 
attitudes  of  these  two  key  groups  are  important  indicators  of 
the  overall  acceptance  of  an  innovation  such  as  CBE;  student 
and  instructor  acceptance  are  essential  for  effective  imple- 
mentation . 

Student  Attitudes 

Short  form  survey.  The  similarities  and  differences 
between  the  PLATO  and  non-PLATO  versions  of  the  short  form 
survey  and  the  scoring  procedures  for  both  are  appropriately 
described . 

Method . The  procedures  for  data  collection  are 
described  concisely.  Mo  further  comment  is  needed. 

Results . Given  the  decision  to  combine  the  individual 
attitud  items  to  form  a single  scale,  the  combinations  chosen 
by  the  authors  for  the  PLATO  version  and  the  MP  version  seem 
appropriate.  The  reliabilities  of  the  resulting  scales  are 
sufficiently  high,  however,  to  suggest  that  the  individual 
item  reliabilities  may  also  be  fairly  large.  If  so,  combin- 
ing the  item  scores  into  a single  scale  score  may  effectively 
"throw  away"  some  information  available  in  the  individual 
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items.  Tatsuoka,  Misselt,  and  Maritz  (1977)  report  a number  of 
analyses  of  the  short  form  survey  response  data  which  treat 
each  item  as  a separate  variable.  In  a similar  vein,  the 
decision  by  the  report  authors  to  base  their  analyses  on  a 
random  selection  of  20  subjects  from  each  condition  does  not 
make  full  use  of  the  available  data.  Rather  than  "throw 
away”  data  from  the  PB  and  CP  conditions,  the  authors  might 
have  used  the  unweighted  means  analysis  or  the  least  squares 
solution  (Kirk,  1968,  pp.  276-282). 

The  first  analysis  reported  was  a comparison  of  the  so- 
called  "PLATO-using"  conditions  (PB  and  CP).  It  used  a scale 
based  on  a combination  of  all  five  short  form  items  and 
showed  a significant  drop  in  attitude  as  the  students  pro- 
gressed through  the  blocks.  The  CP  group  held  slightly  more 
favorable  attitudes  than  the  PB  group,  but  the  difference  was 
not  significant.  Both  groups'  attitudes  remained  positive 
throughout  the  course  even  though  they  were  less  positive  at 
the  end  than  they  had  been  at  the  beginning. 

The  second  analysis  also  used  only  20  subjects  per 
group,  but  it  allowed  comparisons  among  all  three  conditions 
on  the  scale  derived  from  the  three  media  acceptance  items. 

As  before,  there  were  no  significant  differences  among  the 
conditions,  although  the  use  of  all  available  data  would  pro- 
bably have  increased  the  power  of  the  test  for  this  effect 
and  significant  differences  might  have  been  found. 

The  summary  of  open-ended  comments  is  informative 
and  is  consistent  with  our  understanding  of  both  the  student 
population  and  the  CBE  lessons  developed  at  Chanute. 


Discussion . The  hypotheses  suggested  in  this 


section  regarding  the  decline  of  attitudes  over  time  are 
reasonable.  The  authors'  hypothesis  that  student  attitudes 
may  have  been  negatively  influenced  by  technical  packaging 
errors  in  the  CBE  lessons  (e.g.,  typing  and  spelling 
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errors,  nonexistence  or  inability  to  access  "advertised" 
branching  sequences,  execution  errors,  etc.)  is  consistent 
with  results  found  by  R.  A.  Avner  (personal  communication) 
in  another  context.  Avner  correlated  "quality  scale"  ratings 
of  23  lessons  with  student  ratings  of  "helpfulness"  for  learn- 
ing the  presented  material.  The  quality  scale  included 
references  to  technical  errors  so  severe  that  they  prevented 
students  from  completing  a lesson,  as  well  as  minor 
typographical  errors  that  were  only  cosmetically  harmful. 

He  found  a significant  relationship  between  the  two  sets  of 
ratings  (r(21)=.923  with  .95  Confidence  range  of  .825  to 
.967).  The  Chanute  materials  contained  a number  of  these 
technical  problems  during  the  data  collection  period. 

It  is  also  plausible  that  the  PLATO-using  students  were 
subject  to  the  "Hawthorne  effect"  during  this  study.  This 
effect  could  have  caused  an  initial  inflation  in  their 
attitudes  toward  CEE  followed  by  a gradual  decrease  through- 
out the  course.  In  any  event,  the  attitudes  toward  PLATO 
were  more  consistently  positive  than  attitudes  toward 
instructors.  This  consistency  may  be  a real  advantage  for 
CBE--especially  during  periods  of  mobilization  when  experi- 
ienced  instructors  may  be  difficult  to  find. 

Long  form  survey.  The  description  of  the  survey  ques- 
tionnaire is  quite  adequate  for  communicating  its  essential 
features . 

Method . The  factor  analysis  of  the  combined  PB  and  CP 
survey  responses  is  not  reported  in  as  much  detail  as  might 
be  useful  for  independent  interpretation.  Some  additional 
detail  is  provided  in  this  critique. 

The  responses  of  the  200  PLATO-using  students  on  the 
bb-item  survey  were  subjected  to  a principal  components 
analysis  and  the  resulting  components  were  rotated  by  the 
varimax  method.  The  five  factors  that  were  extracted 
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accounted  for  the  following  percentages  of  total  variation: 
30.52,  6.05,  3.95,  3.63,  and  3-21.  Together  they  accounted 
for  47.09  percent  of  the  total  sample  variation. 

The  interpretations  of  the  various  factors  suggested 
by  the  authors  may  be  over-simplified  in  that  the  factor 
labels  apply  properly  only  to  the  aggregate  of  items  whose 
factor  loadings  are  reported  in  Appendix  D;  they  do  not 
apply  so  readily  when  the  loadings  for  the  remaining  items 
are  examine^.  The  "purity"  of  interpretation  implied  by  the 
various  factor  labels  is  attained  through  the  expedient  of 
ignoring  those  items  which  appear  to  be  in  conflict  with  the 
chosen  interpretation  label.  To  be  sure,  the  items  on  each 
factor  which  are  ignored  are  those  which  have  "low"  loadings. 
The  definition  of  "low",  however,  differs  from  one  factor  to 
another;  the  items  the  authors  choose  to  ignore  on  some 
factors  actually  have  higher  loadings  than  those  of  items 
which  were  retained  on  other  factors. 

Results . The  report  authors  computed  "raw"  scale 
scores  on  the  various  factors  for  each  individual  by  summing 
scores  on  those  items  whose  factor  loadings  are  reported 
in  Appendix  D.  Since  the  number  of  items  selected  for 
inclusion  differs  from  factor  to  factor  (because  of  the 
differing  definitions  of  a cutoff  for  loading  values 
described  above),  the  raw  sums  for  each  factor  were  divided 
by  the  number  of  items  to  give  scores  representing  a person's 
"average"  response. 

The  interpretation  of  the  mean  scores  for  each  factor 
scale  (i.e.,  the  discussion  of  the  results  in  Table  24)  seems 
reasonable,  but  it  is  difficult  to  do  an  independent 
interpretation  without  also  being  told  the  scales’  standard 
deviations . 

Two  sets  of  comparisons  were  completed  using  the  mean 

The  first  of  these  was  a 
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scale  scores  described  above. 
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scale-bv-scale  comparison  of  the  two  PLATO-using  groups  whose 
data  were  the  basis  for  the  factor  analysis.  No  significant 
differences  were  found  using  a series  of  independent  groups 
_t-tests  for  the  five  scales.  Given  the  non-orthogonal  nature 
of  the  variables,  it  may  have  been  more  appropriate  to  do  a 
multivariate  analysis  (e.g.,  Hotelling's  T 2 or  MANOVA 
followed  by  discriminant  analysis)  rather  than  a series  of 
univariate  ^-tests  (see  Tatsuoka,  1971). 

The  second  set  of  analyses  was  a series  of  one-way 
univariate  ANOVAs  comparing  the  mean  scale  scores  of  the 
Conventional  PLATO  (CP),  Early  PLATO-Based  (EPB),  and 
Later  PLATO-Based  (LPB)  students.  The  distinction  between 
the  EPB  and  LPB  students  is  that  the  EPB  students  studied 
the  ISD  version  of  the  common  course  segment  before  May,  1975, 
and  the  LPB  classes  studied  the  same  materials  (except  for 
possible  revisions)  after  May  1975.  As  was  true  of  the 
earlier  analyses,  it  would  have  been  preferable  to  have 
applied  a multivariate  technique  (e.g.,  MANOVA)  rather  than 
to  have  done  a series  of  univariate  tests  with  non-orthogonal 
variables.  If  an  overall  multivariate  analysis  were  to  give 
significant  results,  the  exact  nature  of  the  group  differ- 
ences could  be  determined  using  discriminant  analysis. 

Because  of  possible  interrelatedness  of  the  scale  variables, 
the  significant  univariate  ANOVA  result  reported  by  the 
authors  might  be  an  artifact.  Even  taking  this  result  at 
face  value,  it  is  impossible  to  interpret  without  a knowledge 
of  the  group  standard  deviations  on  the  scale  for  factor  1. 

It  is  unfortunate  that  this  information  was  not  reported. 

Another  analysis  of  the  long  form  survey  data  focused  on 
the  preferences  of  the  CP  and  PB  groups  (the  latter  including 
both  EPB  and  LPB  classes)  for  each  of  the  various  media  used 
in  the  course.  Mean  responses  for  these  groups  were  compared 
on  an  item-by-item  basis  using  ^-tests.  The  items  in 
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question  were  items  III-1  to  III-6  in  the  long  form  survey 
(see  Appendix  C).  The  earlier  comments  regarding  use  of 
multivariate  techniques  apply  to  this  set  of  analyses  as 
well.  This  caution  noted,  the  only  comparisons  yielding 
significant  t statistics  were  those  regarding  item  III-2  (PB 
students  viewed  PLATO  lessons  more  favorably  than  CP  students) 
and  item  III-5  (CP  students  were  even  more  favorably  disposed 
toward  laboratory  instruction  than  the  PB  students)  . 

The  graphical  portrayal  of  mean  responses  toward  the 
various  media  shown  in  Figure  5 of  the  AFHRL  report  is  a 
concise  format  for  conveying  the  media  preferences  of  the 
two  groups.  It  would  also  have  been  helpful  to  have 
indicated  the  within-group  variation  in  stated  preference. 

At  any  rate,  one  of  the  most  striking  observations  to  be 
made  from  Figure  5 is  that  the  average  vehicle  maintenance 
student  has  a strong  preference  for  laboratory  activities; 
he  would  much  prefer  to  work  with  his  hands  than  to  engage 
in  learning  activities  that  are  more  cognitive  in  nature. 

Items  III-7  to  III-18  of  the  long  form  survey  (see 
Appendix  C of  the  AFHRL  report)  asked  the  PLATO-using 
students  to  indicate  their  emotional  responses  toward  the 
PLATO  system  as  used  in  their  course.  The  individual 
^-tests  that  proved  significant  and  their  interp>"etation  are 
aptly  discussed  by  the  authors.  Perhaps  more  interesting 
than  these  PB  vs.  CP  comparisons  is  the  graphic 
representation  of  the  mean  response  patterns  for  the  two 
groups  shown  in  Figure  6.  It  is  apparent  that  the  overall 
response  toward  PLATO  was  positive.  Information  on  the 
group  standard  deviations  for  each  item  would  have  been 
helpful  in  determining  the  extent  to  which  the  reported 
means  are  typical  of  the  individual  student  responses. 

The  correlational  analyses  indicating  the  positive 
relationship  between  attitude  and  perceived  reliability  of 
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the  system  are  quite  interesting  and  are  consistent  with 
our  experience  with  other  PLATO  users.  It  is  also 
interesting  to  note,  however,  that  the  attitudes  of  these 
students  (regarding  perceived  reliability  as  well  as  other 
aspects  of  their  experience  with  PLATO)  were  not  related 
to  their  performance  on  block  exams  or  the  special  topical 
test  (see  Tatsuoka,  Misselt  & Maritz,  1977).  Perhaps  one 
reason  interruptions  to  service  did  not  affect  performance 
is  that  student  learning  was  controlled  by  the  mastery  learn- 
ing criterion.  If  a student  happened  to  earn  a low  score  on 
an  end-of-lesson  test  because  of  system  unreliability 
during  study  of  that  lesson,  he  was  routed  back  to  the 
lesson's  starting  point.  The  mastery  criterion  served  as 
insurance  against  ill  effects  of  unreliability  as  well  as 
those  of  other  factors  which  may  hinder  performance.  Hence, 
scores  on  block  tests  were  not  affected  by  system  "crashes". 
It  is  not  known  whether  interruptions  to  service  may  detract 
from  performance  in  non-mastery  settings. 

Discussion . The  commentary  given  by  the  authors  on  the 
attitude  results  is  generally  quite  reasonable.  The 
conclusion  that  the  automotive  maintenance  students  were 
very  accepting  of  PLATO  as  an  instructional  medium  seems  to 
be  well  justified. 

Technical  training  survey.  This  section's  discussion 
of  the  purpose  of  the  technical  training  survey  and  its  12 
scales  is  adequate  for  an  understanding  of  its  application 
in  the  service  test.  More  complete  information  on  its 
development  and  the  results  of  its  use  in  the  broader  con- 
text of  Air  Force  technical  training  presumably  is  forthcom- 
ing in  a separate  publication. 

Method.  No  comment  is  needed. 

Results . Sufficient  usable  data  were  available  to 
perform  4x2  (condition  x administration)  univariate 


repeated  measures  ANOVAs  on  eight  of  the  12  sets  of  scale 


scores.  Although  a multivariate  analysis  technique  might  be 
preferred,  the  authors'  summary  of  the  univariate  results  is 
clear  and  concise.  To  be  more  complete,  the  summary  might 
have  included  tables  of  means  and  standard  deviations  for 
each  administration  to  each  of  the  four  conditions  studied 
Ci.e.,  BL,  NP,  PB,  and  CP). 

The  comparisons  of  the  automotive  maintenance  students' 
attitudes  with  those  of  other  technical  training  students  at 
Chanute  and  within  Air  Training  Command  as  a whole  were 
conducted  by  computing  confidence  intervals  around  the 
observed  mean  on  each  scale  for  the  students  in  each 
condition.  Estimates  of  the  Chanute  and  ATC  population 
standard  deviations  were  used  in  computing  the  various 
confidence  intervals.  If  the  total  sample  mean  for  the 
Chanute  trainees  (n=2484)  or  for  all  ATC  trainees  (n=12666) 
fell  outside  a particular  confidence  interval,  it  was  taken 
as  evidence  that  the  sample  differed  significantly  from 
the  respective  population  on  the  scale  in  question.  Since 
the  estimates  of  the  population  standard  deviations  are  not 
reported,  we  are  unable  to  check  the  calculation  of  the 
various  confidence  intervals.  However,  the  procedure  seems 
to  be  a reasonable  approach  to  the  analysis  of  these  data. 


It  might  be  wished  that  the  authors  had  taken  greater 
care  in  summarizing  the  results  in  terms  of  completeness  and 
accuracy,  but  they  have  reported  the  data  in  sufficient 
detail  (see  Appendixes  F and  G)  that  interested  readers  can 
search  for  other  relationships. 

Discussion . No  further  comment  is  needed. 

Instructor  Attitudes 

Method . The  description  of  the  instructor  attitude 
scale  is  adequate  as  background  for  presenting  the  results. 
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It  would  have  been  helpful  to  have  given  a fuller  descrip- 
tion of  the  sample  of  instructors  since  it  is  apparent  that 
they  differed  with  respect  to  the  duration  of  their  exper- 
ience with  the  PLATO-based  training  system. 

Kesults . It  is  commendable  that  reliability  estimates 
were  computed  for  those  scales  where  it  was  possible  to  do 
so.  Unfortunately,  however,  there  are  some  difficulties  with 
the  statistical  techniques  used  on  these  data.  For  example, 
the  current  description  of  Table  28  appears  to  be  somewhat 
misleading,  as  it  describes  "t-tests  for  the  attitudinal  data 
analyzed  as  repeated  measures  with  correlated  observations." 
This  would  seem  to  imply  that  a series  of  correlated  _t-tests 
had  been  computed  on  the  scaled  responses  for  the  two  admin- 
istrations of  the  survey.  However,  the  analyses  actually 
reported  in  Table  28  were  independent  groups  t-tests.  Since 
there  is  an  overlap  of  10  instructors  who  were  surveyed  in 
both  administrations,  it  is  likely  that  the  assumption  of 
independence  is  violated  in  these  data.  Hence,  the  validity 
of  the  probability  statements  in  Table  28  is  questionable.  A 
more  defenr’hle  procedure  for  assessing  attitudinal  changes 
would  be  to  rcentrate  on  the  responses  of  those  instructors 
who  had  been  surveyed  twice.  A multivariate  analogue  of  the 
correlated  t-test  could  then  be  done  on  the  two  sets  of  scale 
scores . 

Responses  to  the  items  underlying  the  scale  labeled 
"PLATO  affect"  (i.e.,  items  23  to  3^  in  Appendix  H)  were 
examined  more  closely  in  supplementary  analyses.  While  a 
more  complete  description  of  the  "+"  and  "-"  categories 
in  the  Chi  square  analysis  would  be  helpful,  the  Chi  square 
results  confirm  the  earlier  t-test  results  regarding  the 
"PLATO  affect"  scale.  Figure  9 gives  a graphic  portrayal 
of  the  fact  that  attitudes  became  more  negative  at  the  time 
of  the  second  administration  whereas  they  previously  had 
been  generally  positive. 
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Discussion . The  authors'  interpretation  of  the  declin- 
ing attitudes  observed  in  the  survey  responses  seems 
plausible;  a decline  in  perceived  role  importance  could 
result  in  less  favorable  attitudes  toward  CBE.  The  authors' 
suggestions  regarding  increased  instructor  involvement 
appear  to  be  well  founded  and  should  be  investigated  empiri- 
cally in  future  CBE  applications. 

Borne  additional  comments  seem  in  order  regarding  the 
interpretation  of  these  results.  First  of  all,  the  timing  of 
the  questionnaire  administration  should  be  noted.  The 
initial  administration  came  in  February  1975  after  only  a few 
weeks  of  operational  use  of  the  PLATO  system  in  the  revised 
common  course  segment,  and  the  second  came  in  July  of  that 
year.  Although  it  occurred  several  months  into  the 
operational  trials,  the  second  administration  must  still  be 
considered  to  have  been  quite  early  in  the  life  span  of  the 
Phase  II  operation.  Many  instructional  and  coding  problems 
remained  in  the  lessons  (most  of  which  had  not  yet  reached 
Chanute's  validation  criterion)  and  the  group-pacing  of  the 
shred  areas  had  only  recently  been  completed.  In  short, 
there  were  still  a number  of  "bugs"  in  the  newly-revised 
courses  which  may  have  negatively  influenced  the  instructors' 
attitudes  toward  PLATO  in  the  time  frame  of  the  second  survey. 

Another  item  suggesting  that  the  decline  in  instructor 
attitudes  may  have  been  due  to  temporary  factors  is  that 
more  recent  interviews  with  instructors  (Klecka,  1977c) 
revealed  a generally  positive  attitude  toward  PLATO.  It  is 
not  known  how  many  of  the  individuals  interviewed  by  Klecka 
had  been  part  of  the  instructor  force  at  the  time  of  the 
AFHRL  survey.  It  is  possible  that  substantial  "turnover" 
had  taken  place  so  that  some  newer  instructors  learned  a 
"PLATO  role"  without  having  first  become  accustomed  to  a 
more  traditional  role.  Without  any  previous  role  pattern  or 
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expectations,  new  instructors  may  adapt  more  easily  than  did 
the  original  staff.  Perhaps,  however,  the  most  direct  way 
of  testing  the  hypothesis  that  the  attitude  decline  observed 
in  the  AFHRL  survey  of  July  1975  was  due  to  temporary 
factors  would  be  to  administer  that  survey  a third  time 
(i.e.,  to  the  current  group  of  instructors). 

Impact  on  Training  Activities 

Procedure . It  is  highly  commendable  that  observational 
data  were  collected  and  reported  by  the  service  test  evalua- 
tion team,  but  some  aspects  of  the  method  of  gathering  these 
data  are  not  clear  from  the  report's  description.  In  partic- 
ular, it  can  not  be  determined  from  the  report  whether  the 
observation  categories  were  used  in  making  the  observations 
or  whether  they  were  arrived  at  after  the  data  had  been 
collected.  Although  a highly  structured  observation  instru- 
ment would  have  been  impractical,  it  would  be  reassuring  to 
know  that  the  observers  had  at  least  been  instructed  to 
attend  to  some  general  categories  of  behavior  and  that  there 
had  been  some  ground  rules  for  the  classification  and 
reporting  of  observations  to  minimize  observer  bias. 

Results  and  Discussion 

Conduct  of  training.  The  description  of  the  adjustments 
to  the  standard  operating  procedures  (made  necessary  by 
Chanute's  CBE  implementation)  is  enlightening.  The  informal 
"sign-up  sheet"  scheduling  system  seemed  to  work  fairly  well; 
however,  it  is  not  known  whether  it  would  continue  to  be  a 
smooth  and  convenient  way  to  allocate  terminal  time  under 
conditions  of  greater  demand  for  this  resource.  A more 
sophisticated  scheme  may  be  necessary  in  a full-scale 
operational  environment. 
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The  report  indicates  that  the  Chanute  site  occasionally 
suffered  from  shortages  of  computer  memory  space  (known  as 
extended  core  storage  or  ECS).  It  should  be  noted  that  such 
problems  occurred  only  when  ECS  usage  at  Chanute  exceeded 
the  allocation  guaranteed  to  them.  The  two  general  ways 
that  this  problem  can  be  solved  are  (a)  making  more  ECS 
available  for  each  user  (by  adding  expensive  hardware  to  the 
system's  mainframe  or  by  restricting  the  number  of  users  on 
the  system),  and  (b)  reducing  the  amount  of  ECS  required  by 
each  user.  The  AFHRL  report  implies  that  if  PLATO  were  an 
Air  Force  resource,  memory  problems  would  be  eliminated.  In 
order  for  this  to  be  a solution,  restrictions  on  the  number 
of  users  on  an  Air  Force  PLATO  system  could  allow  each  user 
to  occupy  more  memory--probably  with  increased  costs  per 
user.  The  second  approach  to  the  problem  would  require  the 
average  Chanute  lesson  to  be  reduced  to  one-half  or 
one-third  of  its  current  size.  The  latter  solution  would 
consume  slightly  more  computer  disk  storage  (a  relatively 
inexpensive  resource)  but  would  require  a major  modification 
effort  to  partition  existing  lessons  into  a greater  number 
of  smaller  modules.  Future  ECS  problems  could  be  avoided 
by  careful  planning  early  in  the  course  of  a project.  The 
most  viable  approach  would  be  to  keep  the  ECS  requirements  of 
discrete  lesson  segments  under  2000  words. 

Student  interaction.  The  description  of’  student  inter- 
action in  the  traditional  and  CBE  classrooms  is  useful.  It 
affords  a much  fuller  picture  of  the  transactions  in  the 
training  environment  than  could  be  had  through  test  scores 
and  attitude  responses  alone. 

The  classroom  interaction  rate  data  for  the  vehicle 
training  lecture  sessions  is  especially  informative.  Based 
on  data  supplied  by  the  AFHRL  report,  the  following  comparison 
can  be  made  between  the  number  of  questions  answered  by 
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students  in  a PLATO  class  and  in  a lecture  class.  According 
to  this  section,  11  students  answered  77  questions  in  four 

hours,  i.e.,  less  than  two  questions  per  student  per  hour.  By  i 

summing  the  information  in  Tables  17  and  31,  one  can  ' 

calculate  that  PLATO  students  answered  155  questions  (even 
more  when  reviewing  is  counted)  in  an  average  of  369 
minutes,  or  more  than  25  questions  per  student  per  hour. 

We  take  issue  with  some  of  the  comments  made  by  the 
report  authors  in  comparing  CBE  and  conventional  instruction. 

For  example,  their  criticism  that  in  comparison  to  instruc-  i 

tion  by  lecture,  CPE  lessons  are  less  easily  adapted  to  the  | 

needs  of  students  may  be  true,  but  the  arguments  by  which 

they  draw  this  conclusion  are  not  valid.  "Adaptability"  as 

used  in  Chapter  V seems  to  mean  the  ability  to  reshape  the 

delivery  of  material  to  a class.  This  is  rarely  attempted 

in  CBE  because  of  the  availability  of  "individualization", 

meaning  adaptation  for  each  student  rather  than  for  each 

class.  "Adaptability"  also  implies  the  ability  to  halt 

normal  training  when  a class-wide  misunderstanding  is  noted 

and  to  then  re-teach  any  necessary  concepts.  Because  the 

lecture  format  is  not  standardized,  this  adaptability 

fulfills  a very  great  need--it  compensates  for  omissions  or 

inadequate  explanations  by  some  instructors.  With  well- 

tested  standardized  media  presentations,  however,  this 

"total  adaptability"  becomes  unimportant.  Furthermore, 

Chanute  CBE  lessons  were  not,  in  our  opinion,  designed  for 
either  adaptability  or  individualization,  so  that  the  AFHRL 
report  comments  may  be  appropriate  if  applied  to  Chanute  CBE 
lessons,  but  not  necessarily  to  CBE  in  general. 

Because  both  the  PLATO  system  and  the  now-freed-from- 
lecture  instructors  can  gather  data  about  student  questions  , 

as  an  aid  for  revision,  we  do  not  agree  that,  " [ a] ttempting  ] 

to  keep  a record  of  questions  . . . would  be  a difficult,  i 
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if  not  impossible,  task  for  a single  instructor."  Revisions 
based  on  just  this  sort  of  data  can  give  CBE  lessons  a huge 
advantage  over  other  media.  Furthermore,  the  capability  of 
the  PLATO  system  to  collect,  store,  and  organize  data  for 
formative  evaluation  or  individualized  training  is  certainly 
greater  than  that  of  the  typical  classroom  lecturer.  Never- 
theless, the  instructor  who  monitors  a CBE  classroom  is  in  a 
position  to  observe  a number  of  problem  areas  in  the  CRE 
lessons  and,  if  he  has  the  inclination  to  do  so,  can  supple- 
ment and  clarify  the  student  interaction  data  collected 
by  the  system  itself.  Perhaps  the  Chanute  instructors  could 
be  given  some  additional  training  in  techniques  for  gathering 
formative  evaluation  data;  the  recently-added  "TERM-comments" 
feature,  for  example,  could  be  used  quite  effectively  by 
instructors  who  have  observed  areas  of  needed  improvement  in 
the  CBE  lessons. 

Instructional  content.  The  observation  that  students  in 
this  population  were  less  interested  in  theory  or  abstract 
principles  of  engine  operation  than  they  were  in  laboratory 
tasks  confirms  the  result  noted  earlier  from  their  responses 
on  the  long-form  attitude  survey.  We  would  also  underscore 
the  observation  that  the  ability  to  repeat  in  a tireless, 
standardized  way  drills  and  exercises  which  a human 
instructor  would  disdain  and  execute  poorly  is  one  of  the 
most  valuable  assets  of  CBE. 

Instructional  styles.  The  comments  in  this  section  with 
regard  to  the  differing  teaching  styles  of  instructors  and 
the  adjustment  that  students  had  to  make  as  a result  are 
quite  interesting.  It  would  seem  that  this  phenomonon  occurs 
in  other  educational  settings  as  well,  but  it  may  be  more 
acute  in  military  training  environments  where  instructor 
changes  occur  frequently.  That  a similar  adjustment  was 
required  in  going  from  one  CBE  lesson  to  another  was  also  an 
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interesting  observation.  In  this  case,  however,  it  implies 
a lack  of  product  uniformity  standards  during  lesson  develop- 
ment. Had  such  standards  been  employed,  the  lessons  would 
have  been  more  similar  in  format  and  would  have  required 
fewer  adjustments  on  the  part  of  the  students. 

We  would  also  note  that  the  comments  in  this  section 
concerning  the  active  exploration  of  the  characteristics  of 
PLATO  instruction  stand  out  in  strong  contrast  to  the 
soporific  reactions  of  students  to  lecture  presentations. 

Even  the  effort  of  the  students  to  "beat  the  system"  may  be 
viewed  as  positive  when  compared  to  the  lack  of  interest 
exhibited  in  the  lecture  classroom. 

Discussion . The  authors  conclude  in  this  section 
that  "control  of  the  instructional  material  must  be  handled 
very  carefully."  It  cannot  be  assumed,  they  say,  that  all 
students  will  be  motivated  to  do  their  best  work.  That  is, 
given  some  control  over  lesson  sequencing,  there  will  always 
be  a few  students  who  seek  to  "beat  the  system".  Furthermore, 
student  performance  has  not  been  shown  to  be  universally 
better  when  lesson  sequencing  has  been  put  under  the  control 
of  the  student  rather  than  the  computer.  However,  Seidell 
(1975)  has  shown  that  high-performing  students  can  benefit 
from  a degree  of  student  control  over  sequencing.  Learner 
control  may  be  most  effective,  therefore,  when  individualized 
according  to  performance  record.  For  a review  of  research  on 
the  issue  of  learner  control  in  CBE,  see  the  article  by 
Steinberg  (in  press). 

Conclusion . We  have  previously  noted  our  reservations 
regarding  some  of  the  authors'  observations,  but  we  concur 
with  their  conclusions  that  (a)  PLATO  did  not  have  a major 
adverse  impact  that  precluded  its  potential  effective  use  in 
technical  training,  and  (b)  refinements  in  implementation 
procedures  could  reduce  the  negative  impact  of  those  problems 
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that  were  noted.  We  believe  that  many  of  these  refinements 
could  be  made  quite  easily. 

Impact  on  Existing  Air  Force  Regulations  and  Guidance 

Since  we  are  not  part  of  the  Air  Korce  training  environ- 
ment, most  of  the  remarks  of  this  section  are  beyond  our 
sphere  of  direct  experience.  As  close  observers  of  the 
entire  service  test  at  Chanute,  however,  we  do  agree  with  the 
statement  that  the  use  of  the  PLATO  system  as  an  instructional 
resource  did  not  necessitate  great  changes  in  Air  Force 
regulations,  manuals,  and  procedures.  However,  our  view  of 
the  service  test  leads  us  to  underline  the  caveat  that 
future  Air  Force  PLATO  efforts  should  be  conducted  in  the 
environment  of  systematic,  in-depth  instructional  system 
development . 

New  Training  Capabilities 

It  is  true,  as  the  authors  indicate,  that  the  main  goal 
of  Phase  11  was  to  get  operational  training  underway  as 
rapidly  as  possible.  This  orientation  limited  the  use  of 
PLATO  to  presenting  low-level  instruction  in  automotive 
fundamentals  using  instructional  strategies  similar  to  those 
used  in  conventional  programmed  texts.  There  was  little 
opportunity,  in  Phase  II,  to  attempt  applications  which  would 
utilize  the  unique  capabilities  of  the  CBE  medium. 

We  agree  that  the  two  areas  discussed  here  as  potential 
cost-effective  applications  of  the  PLATO  system  (i.e., 
diagnosis/troubleshooting  and  performance  testing)  are  prime 
candidates  for  achieving  cost-effectiveness  while  substanti- 
ally improving  the  quality  of  training.  We  also  agree  with 
the  tentativeness  with  which  these  recommendations  are  put 
forward.  The  successful  development  of  any  CBE  courseware 
depends  upon  a great  many  factors.  Only  by  assessing  the 
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instructional  materials  to  be  produced,  the  environment  in 
which  they  are  to  be  used,  the  capabilities  and  experience 
of  the  instructional  developers,  and  the  management  of  the 
production  effort  can  one  begin  to  predict  the  chances  for 
success.  With  the  right  combination  of  these  factors, 
however,  the  hoped-for  advantages  of  these  CBE  applications 
could  probably  be  realized. 

Overall  Summary  and  Conclusions 

In  this  section  the  authors  summarized  the  important 
results  of  this  extensive  chapter  on  instructional  impact. 

The  major  findings  can  be  summarized  as  follows: 

1 . The  impact  of  PLATO  on  student  attitudes  was 
generally  positive. 

d.  Instructors  reported  positive  attitudes  toward 
PLATO  initially,  but  a second  survey  showed  a decline  in 
attitude.  Based  on  more  recent  attitude  data  (Klecka,  1977c) 
showing  a generally  favorable  attitude  toward  PLATO,  we 
offered  the  hypothesis  that  the  negative  attitudes  found  in 
the  second  AFHRL  survey  were  due  to  temporary  factors. 

i.  PLATO  had  no  significantly  adverse  impact  on  conduct 
of  training  which  would  preclude  its  potentially  cost- 
effective  use.  Furthermore,  refinements  in  implementation 
could  alleviate  much  of  the  negative  impact  that  was 
observed . 

4.  Existing  ATC  regulations  provide  adequately  for  the 
utilization  of  CBE  in  operational  training;  no  major  changes 
in  these  documents  are  needed  to  incorporate  CBE. 

b.  Suggestions  for  additional  PLATO  applications  which 
would  utilize  the  unique  capabilities  of  CBE  were  made. 

We  concur  in  the  conclusion  that  "the  overall  impact 
of  PLATO  has  been  more  positive  than  negative.  There  is  no 
reason  to  believe  that  PLATO  cannot  prove  to  be  an  accept- 
able and  useful  training  medium." 
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VI.  IMSTRUCTIOMAL  MATERIALS  DEVELOPMENT 


This  is  a well-written  chapter  providing  a wealth 
of  detail  about  the  process  and  products  of  the  lesson 
development  efforts.  Although  headings  are  sometimes  a 
bit  misleading,  the  content  is  addressed  in  a logical  and 
orderly  fashion. 


Method 

No  comment  is  needed. 

Results  and  Discussion 

Process  description.  It  is  indeed  accurate  to  say 
that  Phase  II  contrasted  favorably  to  Phase  I in  terms  of 
the  rapid  production  of  effective  lesson  materials.  It 
should  be  pointed  out  that  both  the  methods  used  and  the 
management  staff  changed  substantially  between  phases.  The 
contribution  of  each  to  the  enhanced  efficiency  is  large,  but 
the  relative  importance  of  the  components  is,  unfortunately, 
unknown.  The  model  which  did  emerge  during  Phase  II  was  a 
reasonable  fit  to  the  constraints  of  the  technical  training 
environment.  Future  applications  of  CBE  in  a similar  setting 
would  do  well  to  take  it  as  a starting  point  and  evolv , 
further  refinements. 

The  nature  of  technical  training.  It  is  certainly  true 
that,  as  this  section  states,  the  characteristics  of  the 
courseware  development  effort  in  Phase  II  were  defined  by  the 
nature  of  Air  Force  Technical  training.  In  contrast  to 
higher  education,  where  there  is  considerable  emphasis  on 
development  of  the  higher  levels  of  cognitive  processing  and 
learning  as  an  end  in  itself  (i.e.  "education”),  the  Air 
Force  technical  trainee  is  taught  only  what  he  needs  to  know 
for  apprentice-level  entry  into  a career  field.  It  is 


assumed  that  further  advancements  will  require  further 
preparation  in  the  form  of  Career  Development  Courses  or 
Un-the-Job  Training. 

In  the  technical  training  environment,  o ver-tr aining  is 
costly  and  is  to  be  avoided  just  as  is  under-training . This 
philosophical  orientation  toward  training  (known  as  the 
"lean  approach")  is  realistic  and  effective,  but  it  narrows 
the  range  of  CBE  capabilities  which  can  be  exploited. 

Whereas  in  higher  education  settings  the  simulation  and 
modeling  capabilities  of  CBE  are  usually  considered  to  be 
quite  attractive  (because  they  can  be  used  in  developing 
higher-level  cognitive  skills),  these  features  were  not  seen 
to  be  so  important  in  the  apprentice-level  training 
provided  in  the  Special  Purpose  Vehicle  Courses.  Hence, 
such  sophisticated  applications  as  diagnosis/troubleshooting 
and  performance  testing  (discussed  in  Chapter  V)  were  not 
attempted . 

CM  Lesson  Development 

The  model.  We  agree  that  the  procedural  model 
described  in  this  section  is  a generalized  composite  of  the 
process  actually  used.  Although  individual  deviations  from 
it  were  common,  the  description  provided  by  the  authors 
conveys  the  general  approach  that  was  followed.  The  reader 
should  note  that  the  report  authors  abstracted  this  descrip- 
tion from  a series  of  interviews  with  service  test  personnel. 
No  formal  delineation  of  procedures  had  been  made  by  the  ISD 
staff. 

central  to  the  model  is  the  concept  of  a team  approach. 
In  contrast  to  the  pattern  followed  during  Phase  I in  which 
each  individual  author  was  given  sole  responsibility  for 
all  phases  of  lesson  development,  two  lesson  production 
teams  were  established  in  Phase  II.  Each  team  included 


two  to  three  of  the  original  authors,  but  some  of  the 
development  tasks  were  shared  by  individuals  who  had  more 
specialized  duties.  That  is,  an  instructional  programmer,  a 
computer  programmer  (i.e.,  a "coder"),  and  a subject  matter 
expert  supplemented  the  authors'  efforts.  The  authors  had 
had  previous  experience  with  CBE  by  virtue  of  their  Phase  I 
work,  but  the  other  team  members  (including  the  instructional 
programmers)  had  only  had  experience  with  conventional 
programmed  instruction  and  stand-up  instruction--they  had  not 
previously  developed  lessons  for  a CBE  system. 

As  stated  in  the  AFHRL  report,  the  procedural  model  is 
a description  of  the  processes  followed  in  step  4 (i.e., 
"Plan,  Develop,  and  Validate  Instruction")  of  the  overall 
model  for  instructional  systems  development  (Figure  11). 

The  first  three  steps  of  the  overall  process  had  already 
been  completed  by  the  ISD  team  prior  to  the  start  of  Phase 
II  activities. 

The  nine  steps  in  the  lesson  development  process  are  an 
adequate  portrayal  of  the  procedures  actually  followed. 
General  descriptors  of  the  steps  are: 


1 . 

Lesson  selection. 

2. 

Research , 

3. 

Lesson  planning. 

4 . 

Off-line  lesson  prepara 

tion , 

5. 

On-line  coding. 

b. 

Pre-tryout  review. 

7. 

Student  tryout. 

a. 

Validation  , 

9. 

Finalization. 

A 

more  complete  description 

might  have 

indicated  the 

e 

and  extent  of  individual 

deviations 

from  this 

model 

. so 

would  point  out  that  in 

the  lesson 

planning 

stage 

(step  3)  the  only  optional  media  used  at  Chanute  were 


rear-projected  microfiche  images.  Touch  panels  and  random- 
access  audio  devices  were  not  used  in  any  way  during  Phase 
II,  to  our  knowledge.  Francis  (1976a,  p.  66)  provides  a 
survey  of  the  use  of  peripheral  equipment  at  ARPA/PLATO 
project  sites. 

It  is  not  apparent  from  the  discussion  of  the  model 
that  step  8 was  separated  from  step  7 by  a period  of  up  to 
six  months,  but  the  authors  do  make  this  point  later  in 
the  chapter.  For  more  information  about  lesson  development 
procedures  in  Phase  II,  the  reader  is  referred  to  the  entire 
first  chapter  of  a report  by  Klecka  (1977b). 

Assessment  of  the  model.  This  section  gives  a further 
elaboration  of  Chanute's  courseware  development  model  and 
assesses  its  strengths  and  weaknesses.  While  a more  complete 
analysis  would  be  useful  for  the  sake  of  future  development 
efforts,  the  information  provided  does  give  a general  over- 
view and  is  consistent  with  space  constraints. 

Many  of  the  problems  noted  relate  to  the  implementation 
of  the  model  at  Chanute  and  are  not  disadvantages  inherent  to 
the  model  itself.  The  problems  with  communication,  for 
example,  developed  because  of  the  sudden  manner  with  which 
the  Phase  II  effort  was  initiated.  The  PLATO  authors  were 
put  under  the  direction  of  the  ISO  team  at  a time  when  the 
ISO  group  had  already  completed  most  of  the  initial  planning 
for  the  course  revisions.  The  PLATO  authors  had  not  partici- 
pated in  this  planning  and  the  ISO  group  had  had  little 
previous  experience  with  CBE  (some  of  the  ISD  personnel  had 
attended  a portion  of  the  TUTOR  language  workshop  at  CERL  two 
years  previously,  but  had  never  done  any  direct  CBE  lesson 
development).  Many  basic  issues  in  philosophy  and  orienta- 
tion had  to  be  settled  before  the  effort  could  get  underway, 
and  the  process  was  complicated  by  the  pressure  of  a tight 
production  schedule.  These  problems  in  communication  would 


116 


not  have  been  so  imposing  if  all  groups  had  had  adequate 
training  and  if  all  initial  planning  could  have  been  done 
jointly . 

Because  the  PLATO  authors  were  the  only  team  members 
who  had  had  previous  experience  in  CBE,  they  retained  a 
central  role  in  the  Phase  II  effort.  Their  role  was  not 
drastically  different  under  Phase  II  than  it  had  been  under 
Phase  I.  They  still  had  primary  responsibility  for  most 
aspects  of  lesson  development,  but  relied  on  the  instruc- 
tional programmers  and  subject  matter  experts  as  resource 
persons  and  for  quality  control.  In  future  applications, 
where  more  care  can  be  taken  in  training  and  in  assignment 
of  roles,  a more  efficient  team  organization  should  be 
possible. 

The  development  of  the  student  router,  test  administra- 
tion routines,  and  other  course  management  software  followed 
a different  model  as  indicated  in  the  report.  For  a detailed 
description  of  the  uses  of  the  PLATO  system  as  a management 
tool  at  Chanute,  see  the  second  chapter  of  Klecka  {1977b). 

Appendix  I 

Because  of  the  importance  of  Appendix  I to  the  under- 
standing of  the  conditions  and  procedures  in  the  Chanute 
PLATO  project,  this  critique  devotes  a section  to  comments 
about  it.  The  appendix  consists  of  several  parts,  the  most 
important  of  which  are  the  scenario  and  the  sample  lesson. 

Scenario . The  appended  scenario  is  a livelier  and 
more  picturesque  view  of  the  lesson  development  steps 
described  in  Chapter  VI.  As  such,  it  is  a very  valuable 
addition.  Since  any  scenario  can  consider  only  a limited 
number  of  situations  if  it  is  to  be  of  reasonable  length, 
there  is  the  possibility  that  the  reader  will  gain  an  incom- 
plete perspective  of  the  project.  To  avoid  that  potential 
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problem,  we  have  included  below  several  additional  consider- 
ations that  should  broaden  the  reader's  view. 

Some  authors  suffered  more  from  morale  problems  than  did 
"Mr.  Arthur".  Authors,  frustrated  by  the  constraints  and 
demands  of  their  environment  (e.g.,  terminal  shortages, 
uncertain  project  goals,  lack  of  administrative  support) 
formed  constantly  shifting  alliances  with  and  against  each 
other,  the  ISD  group,  the  evaluation  staff,  and  the  tradi- 
tional course  personnel. 

Because  of  the  author  attrition  noted,  quite  a number  of 
lessons  were  written  by  one  author  and,  after  running  a few 
students,  revised  by  another.  This  may  explain  some  of  the 
inconsistencies  within  lessons  noted  previously  and  in  the 
following  section. 

Sample  lesson.  We  would  encourage  all  readers  to  read 
through  carefully  these  sample  frames  from  a representative 
lesson.  They  exemplify  the  comments  made  in  this  chapter  in 
very  concrete  terms.  From  our  perspective,  the  lesson 
selected  is  typical  of  those  produced  during  Phase  II. 

Strengths  and  weaknesses  are  found,  not  only  between 
lessons,  but  also  wi thin  them.  In  many  cases  the  quality 
of  one  aspect  of  a lesson  varies  widely  from  frame  to  frame 
within  that  lesson.  For  example,  in  the  clutch  lesson  de- 
scribed, the  graphics  are  very  well  done,  but  are  not  always 
available  when  most  needed.  Frame  39  gives  a long  "knee- 
bone-is-connected-to-the-leg-bone"  description  that  desper- 
ately needs  graphic  support  (preferably  with  animation). 

For  a second  example,  the  multiple  choice  questions  within 
the  body  of  the  lesson  are  reasonable  and  integrate  well 
with  the  presentation,  but  the  choice  of  matching  items  in 
the  MVE  includes  both  tangibles  (copper  wires)  and  intangi- 
bles (solid  friction)  as  well  as  singular  and  plural  forms. 
Both  examples  demonstrate  the  variation  in  lesson  quality 
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we  have  observed  in  the  Chanute  materials  and  seem  to  indi- 
cate that  the  potential  for  higher  quality  lessons  lay 
almost  within  the  project's  grasp. 


Courseware  Development  Issues  and  Constraints 

Prior  lesson  development  efforts.  in  this  section  the 
authors  give  a general  description  of  the  activities,  decisions 
and  outcomes  of  the  first  two  years  of  the  project,  i.e.. 

Phase  I.  During  this  period  the  project's  goals  were  either 


undefined  or  were  changing  rapidly — it  wasn't  until  the 
beginning  of  Phase  II  that  a cohesive  curriculum  development 
effort  was  begun.  The  lessons  that  were  produced  in  Phase  I 
were  developed  under  the  "Autonomous  Author"  approach. 

Since  this  basic  model  had  been  used  successfully  at  the 
University  of  Illinois,  CERL  had  advocated  initially  that 
individuals  be  given  full  responsibility  for  developing 
lessons,  i.e.,  to  act  as  independent  authors.  Unfortunately, 
this  approach  was  not  well-suited  to  the  Chanute  situation 
because  several  key  prerequisites  for  its  success  (i.e., 
self-selection  of  authors,  thorough  subject  matter  knowledge, 
relevant  instructional  design  experience)  were  absent.  Later 
efforts  by  CERL  to  foster  a team  approach  were  unsuccessful 
because  of  the  fluctuation  of  project  goals. 

Although,  as  noted,  sel f-tr aining  in  programming 
techniques  occupied  a great  deal  of  time  during  Phase  I,  we 
see  this  less  as  an  actual  need  and  more  as  the  authors' 
attempts  to  "do  something  useful"  during  a period  when  other 
goals  were  rapidly  changing. 

The  diagnosis  that  higher  management  had  little  interest 


and  provided  little  support  to  the  PLATO  project  because  of 


its  research  orientation  is  consistent  with  our  observations 

and  it  seems  to  be  an  important  problem  for  future  sites  as  1 

well.  The  use  of  a novel  medium  such  as  PLATO  suggests  the  j 
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need  for  a research  orientation  to  examine  the  potential  of 
the  medium,  but  a research  orientation  may  seriously 
jeopardize  the  acceptance  of  the  new  medium  and  limit  the 
commitment  toward  it  on  the  part  of  higher-level  admin- 
i strator s . 

The  conclusion  that  Phase  I resulted  in  a fragmented 
non-curriculum  not  conforming  to  civilian  or  military 
standards  for  training  is  congruent  with  the  findings  of  the 
May,  1974,  review  committee. 

Project  management.  This  section  incluaes  an  excellent 
description  of  the  differing  expectations  of  the  iSD  team  and 
the  author  group.  During  the  transition  to  Phase  II,  the 
authors  felt  they  were  being  told  to  sacrifice  quality  for 
high  production;  the  ISD  staff  felt  the  authors  were  unmoti- 
vated. On  many  issues,  the  two  groups'  views  were  vastly 
different.  Initially  the  ISD  staff  hoped  to  produce  PLATO 
lessons  as  quickly  as  programmed  texts  (50hr/hr).  The 
authors  were  used  to  working  at  a rate  about  an  order  of 
magnitude  slower  than  that  (400hr/hr).  Both  estimates 
(obtained  through  CERL  interviews)  were  unrealistic  and 
conflict  on  this  point  was  inevitable.  It  is  significant 
that  problems  such  as  this  were  solved  or  circumvented 
quickly  so  that  courseware  production  could  proceed. 

Other  management  problems  arose  because  the  use  of  CBE 
was  seen  by  course  personnel  as  being  a temporary  experimental 
project  having  little  place  in  operational  training.  Future 
applications  should  seek  to  utilize  CBE  as  an  important 
resource  rather  than  treat  it  as  an  artificial  and  short- 
lived experiment.  If  this  can  be  done,  it  should  be  possible 
to  circumvent  many  management  and  implementation  problems. 

At  the  end  of  this  generally  excellent  section,  the 
AFHRL  report  indicates  that  "[b]eing  linear  and  lean,  the 
lessons  did  not  tax  the  capabilities  of  the  PLATO  system." 
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The  distinction  between  "taxing"  and  "exploiting"  the  vari- 
ous powerful  features  of  the  PLATO  CBE  system  was  confused  fre- 
quently during  Chanute's  history.  Members  of  CERL's  MTC  staff 
and  some  members  of  the  Chanute  author  staff  regularly  advo- 
cated the  exploitation  of  system  features  for  sorting  student 
response  data,  providing  feedback  for  specific  wrong  answers, 
and  for  judging  the  adequacy  of  a wider  variety  of  student 
responses.  This  advocacy  was  often  understood  to  mean  "the 
PLATO  system  has  clever  and  unique  features  which  ought  to 
be  used  because  they  are  so  unusual,  because  they  'give  the 
system  a workout'  (as  if  the  system  somehow  grew  sluggish  if 
not  regularly  exercised),  or  because  the  use  of  these  fea- 
tures justified  and  acclaimed  the  PLATO  system."  In  fact, 
those  encouraging  the  use  of  these  features  saw  instructional 
or  administrative  problems  which  had  potential  solutions 
available  within  the  system  itself.  We  make  this  comment 
to  illustrate  the  nature  and  extent  of  the  problems  caused 
through  differing  concepts  of  the  role  and  capabilities  of 
the  computer.  Hopefully,  these  conflicts  can  be  circumvented 
in  future  development  efforts  through  adequate  training. 

Hardware  and  software  related  constraints.  We  would 
agree  with  the  assessment  that  the  site  was  not  restricted  by 
the  features  and  ease-of-use  of  the  TUTOR  language.  While 
it  is  true  that  some  of  the  authoring  staff  had  limited 
programming  ability,  others  were  quite  capable  and  most 
programming  needs  could  be  handled  without  outside  assist- 
ance. The  presence  of  "coders"  (i.e.,  persons  whose  primary 
role  was  programming  the  computer  rather  than  the  development 
of  teaching  sequences)  on  the  staff  greatly  enhanced  the 
effectiveness  of  the  site. 

For  a variety  of  reasons,  including  the  fact  that  it  was  ! 

one  of  the  first  remote  PLATO  IV  sites,  Chanute  did  suffer 
from  hardware  problems  to  a greater  extent  than  any  other 


ARPA  site.  The  shortages  and  delays  associated  with  terminal 
and  microwave  installation  were  especially  detrimental  to 
Chanute.  Other  resource  shortages  (i.e.,  in  ECS,  disk  file 
space,  and  computing  power)  were  experienced  by  all  users  and 
hindered  the  lesson  development  process  at  Chanute  as  well. 
Fortunately,  most  of  these  problems  have  been  alleviated 
through  various  hardware  enhancements  to  the  central  process- 


ing system. 

Courseware  development.  Chanute  was  one  of  the  earliest 
users  of  microfiche.  As  such,  they  spent  a great  deal  of 
time  during  Phase  I exploring  and  defining  the  limits  of  the 
microfiche  system.  Nevertheless,  it  seems  an  exaggeration  to 
state  that  several  man-months  were  spent  during  Phase  II  in 


refining  this  process.  In  any  case,  the  story  of  Chanute's 
use  of  microfiche  is  so  extraordinary  and  enlightening  that 
Francis  (197ba)  devoted  a sub-chapter  to  describing  it. 

Other  problems  described  in  this  section  relate  to 
the  fact  that  the  authors  were  not  true  subject  matter  ex- 
perts (so  that  they  had  to  spend  time  in  researching  the 
content  to  be  taught)  and  the  ISO  policy  of  limiting  the 
amount  of  student  testing  a lesson  was  to  have  prior  to  its 
operational  use.  These  problems  should  be  addressed  in 
planning  for  future  CDE  development  efforts. 

We  agree  heartily  that  there  was  a problem  in  determin- 
ing the  proper  depth  of  training  for  an  apprentice  mechanic; 
this  resulted  in  some  apparent  inconsistencies  in  the  Chanute 
lesson  materials.  For  example,  the  basic  electricity  lesson 
delves  fairly  deeply  into  theory,  teaching  mechanics 
trainees  about  atoms  and  electrons,  but  less  than  an  hour  is 
spent  in  the  only  lesson  describing  the  complexities  of 
emission  control  systems,  PCV  valves,  and  diverter  valves. 

Part  of  the  confusion  may  lie  in  the  fact  that  the 
Specialty  Training  Standard  (STS)  for  most  knowledge  items 
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Ithe  material  taught  by  CBE)  indicates  that  students  should 
be  taught  to  the  "B"  level.  The  "B"  level  is  defined  as: 

"Can  explain  relationship  of  basic  facts  and  state  general 
principles  about  the  subject.  (PRINCIPLES)"  Although  the 
course  documents  specify  the  depth  of  training  to  be  at  the 
"B"  level,  the  lessons  which  were  produced  seem  to  us  to  be 
aimed  at  the  "A"  level.  The  "A"  level  indicates  a more 
superficial  level  of  training  and  is  defined  by:  "Can 
identify  basic  facts  and  terms  about  the  subject.  (FACTS)" 
Since  field  results  (reported  in  Chapter  III)  indicate 
acceptable  student  performance,  yet  the  teaching  and  testing 
are  not  at  the  "principle"  level  as  defined  (see  Appendix  I 
of  this  report  or  Klecka,  1977a),  it  may  be  that  training  at 
the  "A"  level  is  sufficient.  This  discrepancy  between  train- 
ing levels,  plus  the  problems  noted  in  the  AFHRL  report, 
gave  rise  to  the  "frequently  expressed  concern  about 
insufficient  depth  of  training"  voiced  by  the  authors. 

Authoring.  The  discussion  in  this  section  further 
illuminates  some  of  the  problems  felt  by  authors  under  the 
conditions  of  the  service  test.  Attention  to  ways  of  cir- 
cumventing these  problems  should  be  given  in  future  efforts. 

bince  the  contents  of  the  Lesson  Characteristics, 
Instructional  Strategies,  and  interaction  sections  are  so 
directly  related  to  the  CEKL  publication  by  Klecka  (19YYa), 
and  because  the  elaborations  and  added  emphases  of  Chapter  VI 
are  consistent  with  our  observations,  it  is  not  necessary  for 
us  to  comment  further. 

Instructional  programming  features.  Although  many 
Chanute  lessons  used  microfiche,  it  is  remarkable  that  few, 
if  any,  ever  used  a microfiche  as  part  of  an  end-of-lesson  or 
block  test.  Similarly,  although  students  were  shown  many 
microfiche,  rarely  were  they  asked  any  questions  about  them 
during  the  lesson.  The  effect  of  this  unusual  use  of 
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microfiche  is  that  the  information  presented  via  microfiche 

seems  related,  but  less  important,  than  information  presented  1 

in  other  ways.  It  is  stated  elsewhere  in  the  AFHRL  report  | 

that  students  found  the  use  of  microfiche  images  very  moti-  | 

vating;  it  may  be  that  motivation  was  the  primary  objective  ] 

and  was  achieved. 

Graphics,  typically,  were  more  completely  integrated  | 

into  the  lesson.  As  this  section  of  Chapter  VI  observed,  I 

slow-plotting  graphics  are  disliked  by  students.  In  some  \ 

cases  over ly-complex  screen  graphics  were  used  when  i 

microfiche  might  have  been  employed  more  inexpensively.  In  i 

one  documented  case  (Francis,  1977a),  one  of  the  best  and 
most  hard-working  programmers  at  Chanute  spent  two  weeks 
copying  a single  (slow-plotting)  display  of  an  electrical 
schematic . 

The  last  point  we  would  like  to  make  about  this  section 
is  to  re-emphasize  an  AFHRL  observation  which  we  feel  summar- 
izes our  main  criticism  of  the  lessons.  Students  reportedly 
complained  that  test  questions  did  not  say  things  in  the  same 
way  as  the  text  in  the  body  of  the  lesson.  We  agree  that 
there  is  evidence  to  suggest  that  the  students  were  merely 
associating  words,  not  learning  concepts. 

Student  evaluation.  No  comment  is  needed. 

Lesson  validation.  No  comment  is  needed. 

Validation  techniques.  This  section  also  closely  follows 
Klecka  (1977a)  and  requires  only  one  further  comment.  Among 
the  "technical  errors”  mentioned  at  the  end  of  this  section 
were  a sizeable  number  of  lesson  execution  errors.  Although 
these  errors  are  carefully  documented  by  the  computer, 
several  lessons  which  had  passed  the  validation  criteria 
continued  to  cause  this  major  kind  of  error  as  long  as  nine 
months  after  validation.  This  suggests  a very  narrow  view  of 


"validation” . 


We  concur  with  the  conclusions  that: 


1 . Phase  II  lesson  development  procedures  were  more 
effective  and  efficient  than  those  used  in  Phase  I. 

2.  The  Phase  II  lessons  did  not  exploit  many  CBF  capa- 
bilities because  of  inadequate  resources  and  a lack  of 

; relevant  training. 

1 

j 3.  Role  specialization  contributed  to  increased  effic- 

' iency  of  lesson  production  (i.e.,  we,  too,  expect  a team 

specialist  approach  to  emerge  as  most  appropriate  for 
technical  training  applications). 

4.  The  PLATO  system  provided  ample  flexibility  in 
design  strategies  and  data  collection. 

5.  Sophisticated  CBE  capabilities  may  not  be  necessary 

' for  lessons  having  limited  (i.e.,  low-order)  objectives. 

( 
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Chapter  VII.  MANAGEMENT  OF  THE  PLATO  BASED  COURSE 

We  find  this  whole  chapter  rather  complete  and  percep- 
tive. We  expect  that  future  development  efforts  can  profit 
greatly  from  the  discussion  it  provides.  Due  to  space 
considerations,  our  comments  are  directed  only  at  the  latter 
sections  of  the  chapter. 

Management  of  instructional  material.  Two  items  related 
by  Chapter  VII  deserve  additional  emphasis.  We  agree  with  j 

the  suggestion  that  it  is  not  wise  to  automate  a function  i 

! 

merely  because  it  can  be  done  or  because  it  might  save  a ^ 

small  bit  of  time.  It  indeed  seems  likely  that  student  and  i 

instructor  motivation  and  morale  might  slip  if  certain  ] 

scheduling,  monitoring,  and  controlling  functions  were  left 
entirely  to  the  computer.  The  next  section  of  the  critique 
discusses  some  problems  that  were  pointed  out  to  us  concern- 
ing the  new  role  of  the  instructor. 

The  AFHRL  report  makes  a comment  that  should  be  reflected  j 

back  to  the  operational  problems  noted  and  to  the  results  of  I 

the  cost-effectiveness  evaluation  when  it  observes,  "Ltjhe 
course  was  operated  under  the  assumption  that  PLATO  would  not  j 

reduce  instructor  requirements  and  therefore  the  system 
should  be  utilized  only  in  ways  that  fit  into  traditional 
modes  of  instruction."  Although  such  constraints  are 
sometimes  necessary,  they  force  assumptions  and  allow 
interactions  which  drastically  change  the  environment  of 
implementation.  In  our  view,  the  predominant  attitude  at 
Chanute  was,  "We  will  permit  the  use  of  PLATO  in  those  places 
where  we  can  force  it  into  traditional  roles,"  rather  than, 

"We  will  adapt  our  methods  and  procedures  so  as  to  take 


maximum  advantage  of  the  potential  of  PLATO." 

Instructor  role  in  PLATO-based  training.  The  AFHRL 
report  draws  a somewhat  unpleasant  portrait  of  the  role  of 
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the  instructor  teaching  a block  of  the  automotive  course 
after  it  had  been  modified  for  PLATO  by  ISD.  This  picture 
is,  unfortunately,  supported  by  the  attitude  data  from  Chap- 
ter V.  Two  bases  for  discontent  emerge  from  the  discussions 
in  this  chapter. 

hirst,  the  lessons  themselves  were  a problem.  By  most 
measurements,  these  lessons  were  not  "polished",  i.e.,  they 
req:jired  the  constant  presence  of  a knowledgeable  proctor 
to  assist  students  having  difficulties.  Nagging  problems 
kept  cropping  up  for  the  instructor.  And,  as  described  in 
this  chapter,  efforts  to  correct  those  problems  were  thwarted 
by  an  unfortunately  complex  set  of  procedures.  (Recent  addi- 
tions to  the  PLATO  system  have  provided  a convenient  way  for 
lesson  problems  to  be  reported  to  the  author  staff  without 
entailing  the  "red  tape"  noted  in  the  AFHRL  report  and 
without  causing  major  interruptions  to  the  work  of  the  PLATO 
staff.  Chanute  personnel  have  implemented  this  system  and 
instructors  are  using  it.) 

A second  problem  was  described  by  the  head  of  the  ISD 
group  (personal  communication)  as  the  reason  why  group-pacing 
is  worse  for  instructors  than  either  self-pacing  or  lock-step. 
He  explained  that  because  the  slowest  student  sets  the  rate, 
the  instructor  cannot  sit  down  to  review  and  carefully  teach 
the  slower  students--that  would  generally  mean  the  group 
would  move  at  a still-slower  rate.  Instead  he  can  only  prod 
the  student,  or  sit  with  the  slow  student  and  give  him  the 
answers  so  the  group  can  proceed.  With  self-pacing,  the 
instructor  can  help  the  student  by  individually  tutoring  him, 
by  providing  additional  materials  to  fill  missing  prerequis- 
ites, or  by  restarting  the  student  at  the  point  at  which  he 
found  difficulties.  Lock-step  pacing  usually  means  stand-up 
classroom  instruction  with  the  instructor  as  the  center  of 
attention  and  clearly  in  command.  With  group-pacing,  the 
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I slowest  student  provides  the  basis  for  proceeding  to  the  next 

; topic;  in  many  ways  the  instructor  is  peripheral  or  superflu- 

ous to  the  activity  taking  place. 

The  last  part  of  this  section  points  up  a typical  prob- 
lem of  implementations;  the  conditions  under  which  the 
materials  were  validated  are  not  the  conditions  present  when 
the  materials  are  used  for  operational  training.  There  is  a 
j significant  threat  to  the  integrity  of  the  validation  because 

students  are  now  allowed  or  encouraged  to  take  notes  for  use 
■ ' on  the  end-of-lesson  tests  which  were  used  as  the  validation 

criterion.  While  it  may  be  that  note-taking  is  an  important 
i learning  strategy,  the  effect  of  using  notes  during  the  test 

is  likely  to  be  significant.  Furthermore,  CERL  staff  observ- 
j ing  in  the  Chanute  PLATO  classroom  after  the  lessons  had  been 

validated  saw  students  "getting  answers"  by  questioning  other 
^ students  or  glancing  over  at  a neighboring  terminal.  Test 

j performance  under  these  circumstances  may  differ  from  that 

: under  the  conditions  originally  planned  for  the  lessons 

(i.e.,  no  notes  and  each  student  doing  his  own  work). 

In  general,  the  report  authors  have  provided  valuable 
insight  and  have  made  excellent  recommendations  regarding 
actions  to  be  taken  by  those  in  charge  of  maintenance  and 
^ operation  of  the  special  purpose  vehicle  course.  The  Chanute 

staff  would  do  well  to  heed  these  recommendations. 

Management  of  student  testing.  This  section  accurately 
' depicts  the  various  capabilities  of  the  PLATO  system  and  its 

* utilization  by  the  Chanute  author  and  instructor  staff.  For 

greater  detail  about  instructor  comment  storage  and  retrieval, 
consult  the  chapter  by  Klecka  (1977b)  dealing  with  manage- 
ment applications  at  Chanute. 

General  Conclusions  Concerning  Management 
, The  conclusions  presented  are  an  appropriate  summary 

' I 
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for  this  chapter.  One  sentence  is  especially  interesting: 
the  report  says  "[wjhile  different  techniques  were  necessary 
to  allow  for  PLATO'S  id iosyncrac ies , these  techniques  served 
to  facilitate  efficient  training  rather  than  impede  it." 

This  is  a familiar  finding  for  many  management  tasks  under- 
going computerization.  An  explicit  list  of  examples  of  such 
techniques  would  have  been  especially  valuable. 
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5 Chapter  VIII.  HUMAN  FACTORS  ANALYSIS 

From  the  view  of  the  specialist  in  Human  Factors,  this 
chapter  is  somewhat  disappointing  in  its  lack  of  quantitative 
data.  For  example,  it  would  be  interesting  to  have  specific 
data  for  the  experience  at  the  Chanute  site  on  terminal 
reliability  in  addition  to  the  CERL-provided  data  on  system 

] 

; reliability.  The  reliability  information  provided  by  CERL 

' includes  terminal  reliability  data  averaged  over  the  entire 

. system.  Being  located  only  about  15  miles  from  the  CERL 

hardware  maintenance  facility,  the  Chanute  site's  exper- 
ience may  actually  be  better  than  the  system-wide  average 
reported  by  the  authors. 

From  a practical  standpoint,  however,  it  seems  clear 
that  all  major  considerations  of  systems  design  have  been 
' evaluated  for  potentially  fatal  limitations  and  that  no  such 

limitations  have  been  found.  Given  the  limitations  of 
resources  under  which  the  evaluation  was  performed,  the 
coverage  on  this  point  seems  entirely  appropriate  to  the 
needs  of  the  Services. 


One  point  that  may  need  clarification  is  the  recommenda- 
tion for  indirect  lighting  to  prevent  glare  from  the  face  of 
terminals.  Many  people  include  diffused  light  as  a form  of 
indirect  lighting.  Strictly  speaking,  they  are  correct  since 
the  source  of  light  itself  is  not  visible.  However,  glare 
from  translucent  diffusers  can  be  just  as  troublesome  as 
glare  from  exposed  bulbs  or  tubes.  The  only  sure  cure  for 
the  problem  of  glare  from  display  screens  is  to  insure  that  a 
dark  ceiling  is  the  only  thing  visible  by  reflected  vision  to 
one  seated  at  a terminal.  This  may  be  achieved  by  sharply 
directional  lighting  deeply  recessed  in  the  ceiling,  reverse- 
angled or  vertical  screens,  or  a combination  of  these 
approaches . 


The  experimental  study  reported  in  Chapter  IX  is  the 
only  one  of  seven  studies  planned  for  Phase  III  that  was 
completed.  The  "Historical  Perspective"  section  of  Chapter 
II  indicates  that  it  was  originally  planned  to  refine  each  of 
these  seven  studies  through  three  separate  iterations.  The 
material  reported  in  Chapter  IX  represents  the  one  and  only 
iteration  of  the  Technical  Order  (TO)  study  that  was  com- 
pleted . 


Lesson  Sequence  and  Development  Time 

The  description  given  of  the  games  used  in  the  TO  instruc- 
tional sequence  is  accurate.  It  is  important  to  note  that 
the  strategies  used  in  these  games  are  sufficiently  general 
as  to  allow  them  to  be  used  to  teach  some  other  subject 
matter.  That  is,  there  is  nothing  inherent  in  the  strategies 
which  restricts  their  use  to  the  TO  material. 

It  is  difficult  to  say  how  an  automotive  repairman 
trainee  might  perceive  these  games.  To  CERL  evaluators, 
however,  they  appeared  to  be  either  so  complex  that  learning 
the  rules  and  mastering  the  game  was  more  difficult  than 
mastering  the  material  being  taught  or  that  they  were  lack- 
ing in  motivational  power.  Readers  should  note  the  complex- 
ity of  the  "Wizzard  Game"  described  and  illustrated  in  this 
section  of  the  report.  Not  only  is  the  game  complex  but  the 
rules  lack  any  obvious  relationship  to  tasks  involved  in 
finding  and  using  TOs . 

CERL  personnel  have  verified  that  the  potential  for  the 
use  of  gaming  strategies  is  high  for  TO  instruction.  An 
easily  learned  game  requiring  the  participant  to  classify  TOs 
in  correct  order  was  written  by  CERL  personnel.  This  game 
was  interesting  and  challenging  even  to  those  who  had  no 
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interest  in  TOs . Seve'^al  PLATO  authors  played  this  game 
simply  for  its  recreational  value  and  became  facile  in 
ordering  lists  of  TOs  in  the  process.  This  game  was  not  used 
in  the  TO  sequence  because  it  was  decided  by  those  directing 
the  study  that  it  did  not  meet  the  design  criteria  that  had 
been  specified. 

The  evaluators  in  charge  of  the  study  are  to  be  commended 
for  attempting  to  measure  the  development  time  expended  for 
each  of  the  game  lessons  and  the  mainline  instructional 
lessons  in  the  TO  sequence.  Data  of  this  sort  are  scarce  but 
are  of  obvious  importance  in  answering  questions  of  cost  and 
efficiency  of  production  methods.  Future  development  efforts 
should  attend  to  the  problem  of  collecting  these  types  of 
data . 

Method 

Evaluation  design.  The  use  of  three  cond itions--conven- 
tional  non-PLATO  instruction  (NP),  PLATO-based  version 
(PV),  PLATO  with  adjunct  gaming  (AG;--was  twofold.  One  pur- 
pose as  reported,  was  to  test  the  efficacy  of  gaming  strate- 
gies for  improving  student  performance  and  attitudes.  The 
other  was  tc  see  if  PLATO-based  instruction  by  itself  could 
accomplish  this  result.  Conventional  instruction  in  the  use 
of  TOs  had  long  been  recognized  at  Chanute  as  being  ineffec- 
tive. Thus,  a complete  set  of  PLATO  lessons  was  developed  to 
teach  the  objectives  of  the  conventional  TO  sequence.  The 
games  for  the  adjunct  gaming  condition  were  then  appended 
onto  this  set  of  lessons  by  interspersing  them  among  the 
instructional  modules  as  shown  in  Figure  12. 

The  fact  that  the  games  were  added  onto  a sequence  of 
simple  tutorial  PLATO  lessons,  which  were  essentially  com- 
plete in  themselves,  was  not  the  most  favorable  environment 
in  which  to  determine  the  value  of  gaming  strategies.  In 
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this  context,  they  are  simply  an  appendage  which  would 
certainly  lengthen  the  time  needed  to  complete  the  sequence. 
Moreover,  in  the  event  that  student  performance  was  already 
high  with  the  unadorned  PLATO  TO  sequence,  any  improvement 
coming  as  a result  of  the  games  would  be  difficult  to  detect. 

Measures . The  reporting  of  the  actual  attitude  ques- 
tionnaire would  have  been  useful  for  those  who  might  wish  to 
use  it  or  make  independent  judgments  regarding  its  validity. 
Additional  information  concerning  the  reliability  of  test 
scores  and  attitude  scales  is  presented  in  a later  report 
section . 

Procedures . The  discussion  of  procedures  suffers 
somewhat  from  lack  of  clarity.  This,  no  doubt,  is  largely 
due  to  the  complexities  caused  by  the  differing  schedules 
for  the  general  purpose  and  special  purpose  courses.  Such 
complexities  are  to  be  expected  when  research  must  be  con- 
ducted in  an  operational  environment. 

Results 

Lesson  performance  findings.  There  are  two  main 
difficulties  with  the  analysis  of  procedures  used  for  the 
lesson  completion  time  and  lesson  score  data.  First,  it  is 
probably  inappropriate  to  have  conducted  univariate  tests  on 
each  of  the  seven  modules  for  reasons  stated  earlier  in  this 
critique.  Second,  a more  sensitive  measure  probably  could 
have  been  had  by  combining  lesson  time  data  and  the  lesson 
score  data  across  the  seven  modules.  Given  the  distribution 
of  mean  times  and  scores  shown  in  Table  S'*,  however,  it  is 
not  clear  that  even  the  overall  time  and  lesson  score  meas- 
ures would  have  detected  a difference  between  the  PV  and  AG 
conditions . 

Attitudinal  findings.  The  PV  condition  students  had 
significantly  higher  attitudes  toward  JUs  than  did  the  NP  and 
AG  students. 
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Discussion 

The  games  introduced  into  the  TO  instructional  sequence 
were  simply  appended  to  the  established  lesson  sequence.  One 
would  expect,  therefore,  that  these  games  would  lengthen  the 
completion  times  of  these  lessons.  That  such  a result  was 
found  comes  as  no  surprise.  The  fact  that  the  attitude 
towards  the  subject  matter  was  poorer  for  the  students  in 
both  the  AG  and  the  NP  conditions  than  for  the  PV  condition 
is  interesting.  It  would  seem  that  attitudes  were  improved 
by  the  simple  use  of  PLATO  but  that  this  advantage  was 
negated  by  introducing  games  into  the  sequence.  This  seems 
to  verify  our  opinion  that  the  games  were  amotivational  and 
dull.  Without  being  able  to  view  the  questionnaire  and 
because  of  the  acknowledged  problems  with  the  internal  and 
external  validity  of  the  study,  such  conclusions  can  only  be 
put  forth  tentatively. 

The  AKHRL  evaluation  report  wisely  refrains  from  general- 
izing from  the  study’s  findings  on  gaming.  Had  the  study  t 

been  able  to  proceed,  it  would  have  gone  through  two  more 
iterations  during  which  the  games,  lessons,  and  data  collec- 
tion procedures  would  have  been  refined.  Such  refinement 
might  have  included  integrating  the  games  into  an  instruc- 
tional sequence  instead  of  just  appending  them  to  a sequence 
of  PLATO  lessons.  In  this  way  a more  valid  test  of  the 
efficacy  of  gaming  strategies  could  have  been  obtained. 

finally,  it  is  important  to  note  that  the  PLATO  version 
of  the  TO  lessons  acccmDl ished  much  of  what  it  was  hoped  the 
adjunct  games  would.  These  lessons,  without  the  games,  saved 
student  time  and  produced  more  favorable  student  attitudes 

toward  the  subject  matter.  In  fact,  we  feel  that  the  conclu-  j 

sions  of  this  chapter  overlook  (or  at  least  underemphasize)  J 
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Technical  Orders)  which  had  resisted  several  previous 
attempted  solutions.  The  results  of  using  PLATO  strategies 
without  the  games  (higher  learning  efficiency  with  better 
attitudes)  suggest  that  PLATO  may  be  the  long-sought-after 
solution  for  this  problem.  The  overall  conclusions  of 
Chapter  IX  have  a tone  of  failure,  although  from  the  view  of 
a student,  instructor,  or  course  administrator,  the  results 
are  positive. 
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Chapter  X.  OVERALL  CONCLUGIONi:  A RKCOMMEF^DATIONS 
^vera] 1 Conclusions 

The  overall  conclusions  adequately  summarize  the  findings 
of  the  previous  chapters  and  contain  no  surprises.  In  their 
brevity,  they  overlook  a few  problems  (such  as  declining 
instructor  attitudes)  and  a few  opportunities  (such  as  the 
potential  for  computer-aided  test  delivery),  but  concisely 
remind  the  reader  of  the  salient  outcomes.  They  are  restated 
here  (in  abbreviated  form)  for  the  convenience  of  readers 
who  do  not  have  access  to  the  AFURL  report. 

1.  The  PLATO  system  was  an  effective  medium  for  present- 
ing task-related  cognitive  materials. 

?.  The  Chanute  PLATO  application  was  not  cost-effective 
when  compared  to  courses  employing  less  sophisticated  indi- 
vidualized media  to  perform  similar  functions  in  this 
application.  However,  as  shown  by  our  comments  on  Chapter 
IV,  the  Chanute  application  may  have  approached  cost- 
effectiveness  in  spite  of  the  fact  that  it  may  not  have 
required  the  full  capabilities  of  CDE. 

3.  PLATO  did  not  have  a large  impact  on  the  main  fea- 
tures of  the  training  environment;  it  was  acceptable  to  stu- 
dents and  instructors  and  it  was  compatible  with  existing 
organizational  structures  and  controls.  However,  little  in 
the  way  of  new  training  capabilities  was  demonstrated. 

Due  to  the  complexity  of  the  courseware  preparation 
process,  greater  efficiency  resulted  from  a team  approach 
employing  role  specialties  than  from  the  single  instructor/ 
author  approach. 

5.  The  Chanute  effort  was  limited  in  attempting  addi- 
tional applications  by  constraints  in  management,  resource 
acquisition,  personnel  training  and  experience. 
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6.  Some  management  difficulties  were  experienced 
because  the  PLATO  system  was  not  an  Air  Force  resource. 
Adjustments  were  necessary  in  the  areas  of  training  activity 
scheduling  and  course  instructional  design  selection. 

7.  Few  usage  problems  were  present  in  terms  of  student 
interaction  or  courseware  authoring.  PLATO  was  found  to  be  a 
reliable  tool  once  system  expansion  and  experimentation  was 
moderated . 

Conclusion  #6  requires  a comment.  It  implies  that 
certain  management  difficulties  caused  by  resource  constraints 
could  be  eliminated  by  Air  Force/military  control  of  a PLATO 
system.  Although  maintenance  time,  for  example,  could  be 
shifted  to  accommodate  Chanute's  needs,  adverse  effects  might 
occur  for  other  users  (military  and  civilian)  in  other  time 
zones.  Nevertheless,  it  is  realistic  to  expect  that  any 
agency  large  enough  to  support  its  own  system  would  want  to 
do  so  in  order  to  gain  control  of  scheduling,  maintenance, 
and  resource  allocation  parameters. 

Recommendations  Specific  to  Chanute 

Chanute  CBE  operations.  The  seven  recommendations  on 
this  topic  are  restated  here  in  abbreviated  form: 

1.  In  this  recommendation  the  authors  state  that 
"PLATO  applications  should  be  selectively  expanded  to  other 
course  areas  where  CAI  capabilities  might  be  used  to  solve 
training  problems,  increase  instructional  efficiency,  reduce 
training  resource  requirements,  etc."  They  also  suggest  roles 
for  specific  organizational  units  at  Chanute  and  advocate  the 
development  of  a detailed  plan  to  guide  expansion  of  PLATO 
appl ications . 

2.  The  authors  recommend  the  self-pacing  of  the  shred 
areas  of  the  SPVC  to  take  advantage  of  CBE's  individualized 
nature,  limit  the  need  for  enrichment  materials  and  provide 
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for  more  efficient  use  of  available  terminals.  Although  this 
recommendation  could  lead  to  additional  time  savings,  it 
would  also  entail  purchase  of  additional  training  equipment. 

Nevertheless,  it  seems  useful  to  prepare  an  analysis  of 
projected  costs  and  savings. 

i.  The  authors  recommend  that  terminal  utilization  be 
maximized  by  shifting  surplus  terminals  to  other  courses  to 
enable  exploitation  of  cost-effective  applications. 

4.  The  report  writers  state  that  "Direct  communication 
between  authors  and  instructors  should  be  promoted  to 
facilitate  application  of  PLATO,  and  increase  instructional 
efficiency."  Perhaps  increased  use  of  the  recently  added 
"TERM-comments"  and  "lesson  notes"  features  of  the  PLATO 
system  will  help  meet  this  need. 

b.  This  item  is  a recommendation  to  install  indirect 
lighting  in  the  CBE  classroom  to  eliminate  problems  of  glare. 

As  indicated  in  our  critique  of  Chapter  VIII,  directed  beams 
of  vertical  light  are  preferable  to  indirect  light  in  this 
case . 

6.  This  item  recommends  a reduction  in  the  instructor 
force  in  the  common  course  segment  of  the  SPVC  or  some  use  of 
less  expensive  instructor  aid  personnel. 

Some  additional  remarks  seem  in  order  regarding  recom- 
mendation //I.  Though  we  don't  have  the  appropriate  back- 
ground to  comment  on  the  roles  suggested  for  the  various 
organizations  noted,  we  concur  with  the  general  approach.  We 
agree  heartily  that  "PLATO  applications  should  be  selectively 
expanded  to  other  course  areas  where  CAI  capabilities  might 
be  used  to  solve  training  problems."  We  believe  that  manage- 
ment and  instructor  acceptance  of  PLATO  might  have  been 
significantly  enhanced  if  it  had  been  introduced  as  a tool  to 
solve  a training  problem  perceived  by  staff  in  both  groups. 

Similarly,  the  effectiveness  benefits  might  have 
outweighed  the  costs  if  the  chosen  course  had  contained 
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unsolved  problems  relating  to  training  effectiveness  or  train- 
ing expense.  Chanute  attempted  to  squeeze  higher  efficiency 
from  a training  program  which  was  already  reasonably  opti- 
mized. The  sophisticated  and  expensive  technology  that  they 
used  to  do  this  required  that  they  realize  substantial  gains. 
Chanute  staff  should  not  be  criticized,  however,  for  the 
choice  of  a course  to  implement  PLATO,  nor  for  their  decision 
to  use  it  as  the  central  medium  for  teaching.  The  many 
constraints  on  them,  plus  the  conventional  wisdom  of  that 
time  dictated  these  factors.  However,  in  a document  making 
recommendations  for  future  projects  and  sites,  it  is  very 
important  to  stress  the  application  of  CBE  for  attacking 
unsolved  training  problems  rather  than  as  the  "medium  of 
preference"  for  implementing  all  new  instructional  materials. 

Development  of  CAI  materials.  Five  recommendations 
regarding  courseware  development  and  validation  are  offered. 
They  are  restated  here  for  the  reader’s  convenience.  Accord- 
ing to  the  authors: 

1.  The  PLATO  microfiche  procurement  process  should  be 
analyzed,  then  improved  for  greater  efficiency.  Toward  this 
end,  a new  microfiche  camera  is  being  developed  at  CERL. 

2.  Redesign  of  the  microfiche  projector  might  eliminate 
focus  and  registration  problems. 

3.  CBE  courseware  should  be  developed  through  a team 
approach  using  specialists  who  have  overlapping  knowledge  of 
other  team  member's  roles  in  the  lesson  development  process. 

4.  A training  program  should  be  provided  for  individ- 
uals who  perform  courseware  development  functions.  This  pro- 
gram would  emphasize  instructional  programming  concepts  for 
preparing  individualized  materials. 

5.  Instructor  and  supervisor  involvement  should  be 
increased  in  the  review  and  revision  of  PLATO  courseware. 
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We  concur  that  a training  program  for  instructional 
design  is  needed  for  courseware  developers  (//4).  We  also  feel 
that  training  for  the  managers  of  the  developers  is  needed. 
When  the  Chanute  project  was  started,  no  such  training  was 
available;  in  fact,  the  knowledge  base  from  which  to  train 
staff  was  largely  incomplete.  V/ith  the  experience  gained 
from  the  implementation  of  PLATO  IV,  it  is  now  reasonable 
to  begin  this  task. 

It  is  not  clear  whether  the  order  of  the  recommendations 
indicates  their  perceived  importance.  We  would  rank  the 
suggestions  for  an  instructional  design  training  program 
(#4),  a team  approach  to  courseware  development  (#3)  and 
instructor/supervisor  involvement  (#5)  ahead  of  the  sugges- 
tions about  microfiche  redesign. 

PLATO  Applications 

Suggestions  for  how  to  use  PLATO  in  future  projects  are 
some  of  the  most  important  results  to  be  delivered  by  an 
implementation/research  project  such  as  this.  The  applica- 
tions suggested  by  the  authors  appear  to  have  potential  for 
cost-effectiveness.  A more  extensive  list  of  potentially 
cost-effective  uses  for  CBE  is  contained  in  a list  of  CBE 
site  management  guidelines  (Francis,  1977b). 


Some  Final  Remarks 


The  comments  by  the  authors  in  this  section  are  a 
fitting  summary  to  a well-written  report  on  an  extensive 
undertaking.  We  would  like  to  add  a few  general  remarks. 

Readers  of  this  (and  other)  reports  of  the  use  of  CBE 
systems  should  consider  that  the  results  obtained  are  rather 
sensitive  to  the  conditions  of  the  implementation.  Variations 


in  the  attitudes  of  the  instructors. 
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lessons,  student-pacing  modes,  and  the  integration  of  CBE 
materials  with  instruction  presented  by  other  media  can 
strongly  influence  outcomes  and  results. 

Although  some  of  the  situations  described  in  the  report 
are  necessarily  related  to  conditions  found  only  at  Chanute 
or  only  with  the  SPVC,  in  general,  the  problems  encountered 
are  not  uncommon  or  improbable,  nor  were  the  solutions  used 
atypical  or  unlikely  to  be  tried  elsewhere. 

We  would  like  to  take  this  opportunity  to  again  commend 
the  project  evaluators  and  the  writers  of  this  report  for  a 
significant  accomplishment  of  a difficult  task.  One  of  the 
main  benefits  of  the  Chanute  research  is  the  extensive  data- 
base it  generated.  Supplementary  analyses  of  this  data  are 
underway  at  CERL  and  are  expected  to  lead  to  additional 
reports  on  CBE  and  related  topics. 

V/hile  we  concur  with  the  third  of  the  overall  conclu- 
sions of  the  AFHRL  report  (i.e.,  "[l]ittle  in  the  way  of  new 
training  capabilities  was  demonstrated"),  we  suggest  that 
readers  of  the  AFHRL  report  not  view  the  Chanute  experience 
a.3  an  example  of  how  training  can  be  structured  around  a CBE 
system  to  take  maximum  advantage  of  its  capabilities,  but  as 
an  example  of  how  PLATO  can  be  incorporated  into  a traditional 
military  training  environment  without  causing  changes  in 
basic  routines. 
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